Resilience runbooks

These are the canonical resilience procedures. The flat operations route keeps old headings as compatibility links. A live session or a liveness response is not proof that a new authorization, tunnel, or restored cluster is healthy.

Connectors with a single egress

Procedure gate

Gate	Contract
Classification	Planned change
Prerequisites	A Proxy route, Authority route, approved firewall change, and a Connector with authority_transport = "tunneled", "direct", or "fronted" selected explicitly.
Owner	Network and Connector operators.
Approval	Network change approval before removing direct Authority egress.
Backup point	Save the Connector config and current firewall rules before the cutover.
Command or check	After the firewall change, from the Connector host run nc -z -w 3 "$AUTHORITY_FRONT" "$AUTHORITY_PORT" and nc -z -w 3 "$AUTHORITY_FRONT" "$ENROLL_PORT"; inspect the Proxy tunnel and request-gate metrics.
Expected status and exit	In a tunnel-only design both direct checks fail with a non-zero exit, while the reverse tunnel and the Proxy relay path are available. A Connector request that cannot obtain an Authority decision is denied and is not forwarded.
Stop condition	Stop if a supposedly tunnel-only Connector can still reach a direct Authority path, or if the tunnel is down and new sessions are being treated as allowed.
Recovery	Restore the prior firewall rule, inspect relay and reconnect metrics, and verify the configured route before retrying.
Rollback	Restore the prior authority_transport and firewall rules together.
Escalation	Network owner for routing, Proxy owner for relay, Authority owner for decision availability.

The reverse tunnel is the data plane. tunneled carries the Authority control plane inside it; direct and fronted use their configured egress routes. A Kubernetes request, database statement, model call, web request, or MCP request can require a per-request Authority decision, so existing data sessions do not prove a new session is authorized.

AUTHORITY_FRONT="<authority-front>"
AUTHORITY_PORT="<authority-control-port>"
ENROLL_PORT="<authority-enroll-port>"
nc -z -w 3 "$AUTHORITY_FRONT" "$AUTHORITY_PORT"
nc -z -w 3 "$AUTHORITY_FRONT" "$ENROLL_PORT"

The check is intentionally performed from the Connector host. A configured mode that leaves the direct path routable proves nothing.

Authority failover

Procedure gate

Gate	Contract
Classification	Incident action
Prerequisites	Authenticated Authority membership or configured primary/fallback addresses, a shared persistent store, Proxy and Connector logs, and a controlled client.
Owner	Authority and Proxy incident operators.
Approval	Incident authority before changing endpoint order or directing traffic manually.
Backup point	Preserve the store, audit export, and current endpoint configuration before intervention.
Command or check	Inspect Proxy relay outcomes and run gdsgate --config <operator-config> authority nodes list --health; then perform a controlled client request through the surviving endpoint.
Expected status and exit	Ready followers forward leader-dependent calls to the fenced leader. The Proxy discovers a surviving route after failure; an uncertain state-changing call is not replayed. A tunnelled Connector's next control stream uses the current route. Fresh access recovers after lease takeover; no live lease means new leader-dependent access is refused.
Stop condition	Stop admission if no endpoint serves, audit shipping does not recover, or authority audit verify fails.
Recovery	Confirm store lease and leader ownership, find Authority pool moved to another instance in the Proxy log, and watch relay and reconnect counters return to baseline.
Rollback	Restore the previous endpoint order only after the failed endpoint is understood. Do not split the store or create a second authority history.
Escalation	Database owner for the shared store, Authority owner for leadership, Proxy/network owner for relay reachability.

The Proxy creates lazy channels and does not hold an active socket to every Authority. A tunneled Connector has no Authority address of its own; the Proxy relays each control-plane stream using its current Authority route. A ready follower can forward it to the leader. A long-lived tunnel does not pin the Connector to a stale leader. Every instance serves registration, token-less renewal, workload issuance and administrative commands (calls by instance); the Proxy relays a renewal to the next endpoint when one refuses the connection.

While no instance holds the lease, the audit rows of those calls wait in audit_outbox, and gdsgate_audit_outbox_oldest_age_secs grows. The new leader seals them on its first pass. Every instance keeps watching the queued rows it has seen through the failover, and reports one that leaves the queue without reaching the chain (queued rows). A row removed while no instance runs is not reported.

The surviving instances decide by the store's active policy version within 10 s and hold the stored CA state within 5 s of a change, including one committed by the failed instance just before it stopped. Check gdsgate_policy_version_applied against gdsgate_policy_version_stored and gdsgate_ca_bundle_stale on each instance (policy and CA reload).

A CA rotation the failed leader started is resumed by the new leader from the stored phase, after the rest of the stored propagation or retire wait. A write the former leader makes after the takeover is refused by the store.

Backup and disaster recovery

Procedure gate

Gate	Contract
Classification	Planned change
Prerequisites	An approved backup destination, store credentials supplied outside committed files, external audit keys, node state directories, and a restore test window.
Owner	Database, Authority, and security custody owners.
Approval	Backup and retention approval before changing the schedule or destination.
Backup point	Take a paired store and key backup before every migration, restore, or destructive maintenance.
Command or check	Use the backend-native backup command, such as pg_dump --format=custom --file "$BACKUP_FILE" "$STORE_URL" for PostgreSQL, or copy a quiesced SQLite store. Run sha256sum "$BACKUP_FILE", restore into an isolated store, and run gdsgate --config <restore-config> authority audit verify.
Expected status and exit	The backup command and checksum exit 0; the restore copy opens and the audit verifier exits 0 with its external key.
Stop condition	Stop if a key is missing, a backup is unverified, the store is in-memory, or a restore cannot verify the chain.
Recovery	Retain the failed backup for investigation, make a new backup, and do not delete the last known-good set.
Rollback	Restore the prior store, keys, and node state as a paired set.
Escalation	Security for key custody, database owner for store integrity, Authority owner for service recovery.

Back up Authority's state store, which contains the transport CA, SSH CAs, registry, catalog, request state, and audit chain. Back up both keyrings of [security.integrity] and their custody records, each separately from the store, and any key file the deprecated [audit].*_key_path parameters still name. Keep anchor checkpoints and exported JSON lines off the Authority host. Persist each node's state_dir, including Connector SSH host-key material. A database-only backup cannot prove the same anchored history after restore.

With HA keyring sync enabled, include each Authority's entire configured ha.keyring_sync.state_dir in that same backup generation. It holds the local root (root.key or root.enc), independent receiver.key, snapshot.enc, revision watermarks and any pending installation journal. The old seed keyring files alone do not represent synchronized live material. Keep each directory associated with its original ha.owner; never copy one member's receiver to another. Preserve the matching configuration, local provider credentials and access to any wrapping KMS key. See Authority recovery.

Pause CA, ring and distribution-root transitions while taking the final recovery set. A PostgreSQL dump is transaction-consistent for the database; it does not atomically capture private files on several hosts. Quiesce the changing fleet for the final paired set and rehearse restore in an isolated environment before accepting it. Include Connector recording buffers that have not reached central storage.

Restoring a cluster

Procedure gate

Gate	Contract
Classification	Incident action
Prerequisites	An isolated restore host, approved store and key backups, the matching verified binary, closed traffic, and an operator with Authority recovery authority.
Owner	Authority incident commander with database and security owners.
Approval	Incident authority before restoring or opening traffic.
Backup point	Preserve the failed cluster and every candidate backup before restore.
Command or check	Restore the store and external files, run gdsgate --config <authority-config> authority store migrate --check, run gdsgate --config <authority-config> authority audit verify, start Authority, Proxy, and Connectors with their persisted state, then run HTTP readiness and a controlled session check.
Expected status and exit	Migration check is current, verification exits 0, /healthz is 200, /readyz becomes 200 only after each role sets readiness, node health and catalog match the backup, and a controlled session opens and closes.
Stop condition	Stop before opening traffic on any migration, chain, key, readiness, policy, catalog, or controlled-session failure.
Recovery	Keep traffic closed, return to the isolated host, select the last trusted paired backup, and repeat verification.
Rollback	Stop restored services and restore the previous paired store, external keys, node state, and verified binary. Do not downgrade migration rows manually.
Escalation	Incident commander, security for chain/key mismatch, database owner for restore failure, service owner for readiness.

Restore the store and external audit material at their configured paths. The Authority resumes its transport CA from the store, and clients with persisted trust anchors can continue to verify existing node identities. Start Proxy and Connectors with their persisted state directories. Clients reuse cached identity tokens until expiry and then sign in again. Treat migrations as forward-only: an older binary refuses a store carrying versions it does not know.

For synchronized Authority HA, restore the matching per-owner root, receiver, cache and journals before startup. A database-only rollback can be refused by the local watermark; deleting that watermark does not establish a trusted restore. Verify keyring revisions, root epoch and /readyz on every restored Authority, then test fresh access, denial, audit and recording delivery through the public entry. Existing sessions are not restored.