Resilience runbooks

These are the canonical resilience procedures. The flat operations route keeps old headings as compatibility links. A live session or a liveness response is not proof that a new authorization, tunnel, or restored cluster is healthy.

Connectors with a single egress

Procedure gate

Gate	Contract
Classification	Planned change
Prerequisites	A Proxy route, Authority route, approved firewall change, and a Connector with authority_transport = "tunneled", "direct", or "fronted" selected deliberately.
Owner	Network and Connector operators.
Approval	Network change approval before removing direct Authority egress.
Backup point	Save the Connector config and current firewall rules before the cutover.
Command or check	After the firewall change, from the Connector host run nc -z -w 3 "$AUTHORITY_FRONT" "$AUTHORITY_PORT" and nc -z -w 3 "$AUTHORITY_FRONT" "$ENROLL_PORT"; inspect the Proxy tunnel and request-gate metrics.
Expected status and exit	In a tunnel-only design both direct checks fail with a non-zero exit, while the reverse tunnel and the Proxy relay path are available. A Connector request that cannot obtain an Authority decision is denied rather than forwarded.
Stop condition	Stop if a supposedly tunnel-only Connector can still reach a direct Authority path, or if the tunnel is down and new sessions are being treated as allowed.
Recovery	Restore the prior firewall rule, inspect relay and reconnect metrics, and verify the configured route before retrying.
Rollback	Restore the prior authority_transport and firewall rules together.
Escalation	Network owner for routing, Proxy owner for relay, Authority owner for decision availability.

The reverse tunnel is the data plane. tunneled carries the Authority control plane inside it; direct and fronted use their configured egress routes. A Kubernetes request, database statement, model call, web request, or MCP request can require a per-request Authority decision, so existing data sessions do not prove a new session is authorized.

AUTHORITY_FRONT="<authority-front>"
AUTHORITY_PORT="<authority-control-port>"
ENROLL_PORT="<authority-enroll-port>"
nc -z -w 3 "$AUTHORITY_FRONT" "$AUTHORITY_PORT"
nc -z -w 3 "$AUTHORITY_FRONT" "$ENROLL_PORT"

The check is intentionally performed from the Connector host. A configured mode that leaves the direct path routable proves nothing.

Authority failover

Procedure gate

Gate	Contract
Classification	Incident action
Prerequisites	Ordered [endpoints].authority_failover addresses, a shared persistent store, Proxy and Connector logs, and a controlled client.
Owner	Authority and Proxy incident operators.
Approval	Incident authority before changing endpoint order or directing traffic manually.
Backup point	Preserve the store, audit export, and current endpoint configuration before intervention.
Command or check	Inspect Proxy relay outcomes and run gdsgate --config <operator-config> authority list-nodes --health; then perform a controlled client request through the surviving endpoint.
Expected status and exit	The Proxy lazily tries the next Authority endpoint after Unavailable; successful use becomes sticky for the next call. Follower writes are refused and retried at the next endpoint. A short burst of unavailable calls during leadership movement is expected.
Stop condition	Stop admission if no endpoint serves, audit shipping does not recover, or a follower is receiving writes.
Recovery	Confirm store lease and leader ownership, let the Proxy pool rediscover the leader, and watch relay and reconnect counters return to baseline.
Rollback	Restore the previous endpoint order only after the failed endpoint is understood. Do not split the store or create a second authority history.
Escalation	Database owner for the shared store, Authority owner for leadership, Proxy/network owner for relay reachability.

The Proxy creates lazy channels and does not hold an active socket to every Authority. A tunneled Connector has no Authority address of its own; the Proxy relays each control-plane stream to the endpoint its pool currently uses. A long-lived tunnel does not pin the Connector to a stale leader. Registration and token-less renewal through proxy_join are primary-authority paths and may retry until the primary is leader again.

Backup and disaster recovery

Procedure gate

Gate	Contract
Classification	Planned change
Prerequisites	An approved backup destination, store credentials supplied outside committed files, external audit keys, node state directories, and a restore test window.
Owner	Database, Authority, and security custody owners.
Approval	Backup and retention approval before changing the schedule or destination.
Backup point	Take a paired store and key backup before every migration, restore, or destructive maintenance.
Command or check	Use the backend-native backup command, such as pg_dump --format=custom --file "$BACKUP_FILE" "$STORE_URL" for PostgreSQL, or copy a quiesced SQLite store. Run sha256sum "$BACKUP_FILE", restore into an isolated store, and run gdsgate --config <restore-config> authority verify-audit.
Expected status and exit	The backup command and checksum exit 0; the restore copy opens and the audit verifier exits 0 with its external key.
Stop condition	Stop if a key is missing, a backup is unverified, the store is in-memory, or a restore cannot verify the chain.
Recovery	Retain the failed backup for investigation, make a new backup, and do not delete the last known-good set.
Rollback	Restore the prior store, keys, and node state as a paired set.
Escalation	Security for key custody, database owner for store integrity, Authority owner for service recovery.

Back up Authority's state store, which contains the transport CA, SSH CAs, registry, catalog, request state, and audit chain. Back up every external audit key and custody record: anchor_key_path, approval_key_path, revocation_key_path, delegation_key_path, and inventory_key_path. Keep anchor checkpoints and exported JSON lines off the Authority host. Persist each node's state_dir, including Connector SSH host-key material. A database-only backup cannot prove the same anchored history after restore.

Restoring a cluster

Procedure gate

Gate	Contract
Classification	Incident action
Prerequisites	An isolated restore host, approved store and key backups, the matching verified binary, closed traffic, and an operator with Authority recovery authority.
Owner	Authority incident commander with database and security owners.
Approval	Incident authority before restoring or opening traffic.
Backup point	Preserve the failed cluster and every candidate backup before restore.
Command or check	Restore the store and external files, run gdsgate --config <authority-config> authority migrate --check, run gdsgate --config <authority-config> authority verify-audit, start Authority, Proxy, and Connectors with their persisted state, then run HTTP readiness and a controlled session check.
Expected status and exit	Migration check is current, verification exits 0, /healthz is 200, /readyz becomes 200 only after each role sets readiness, node health and catalog match the backup, and a controlled session opens and closes.
Stop condition	Stop before opening traffic on any migration, chain, key, readiness, policy, catalog, or controlled-session failure.
Recovery	Keep traffic closed, return to the isolated host, select the last trusted paired backup, and repeat verification.
Rollback	Stop restored services and restore the previous paired store, external keys, node state, and verified binary. Do not downgrade migration rows manually.
Escalation	Incident commander, security for chain/key mismatch, database owner for restore failure, service owner for readiness.

Restore the store and external audit material at their configured paths. The Authority resumes its transport CA from the store, and clients with persisted trust anchors can continue to verify existing node identities. Start Proxy and Connectors with their persisted state directories. Clients reuse cached identity tokens until expiry and then sign in again. Treat migrations as forward-only: an older binary refuses a store carrying versions it does not know.