Resilience runbooks¶
These are the canonical resilience procedures. The flat operations route keeps old headings as compatibility links. A live session or a liveness response is not proof that a new authorization, tunnel, or restored cluster is healthy.
Connectors with a single egress¶
Procedure gate¶
| Gate | Contract |
|---|---|
| Classification | Planned change |
| Prerequisites | A Proxy route, Authority route, approved firewall change, and a Connector with authority_transport = "tunneled", "direct", or "fronted" selected explicitly. |
| Owner | Network and Connector operators. |
| Approval | Network change approval before removing direct Authority egress. |
| Backup point | Save the Connector config and current firewall rules before the cutover. |
| Command or check | After the firewall change, from the Connector host run nc -z -w 3 "$AUTHORITY_FRONT" "$AUTHORITY_PORT" and nc -z -w 3 "$AUTHORITY_FRONT" "$ENROLL_PORT"; inspect the Proxy tunnel and request-gate metrics. |
| Expected status and exit | In a tunnel-only design both direct checks fail with a non-zero exit, while the reverse tunnel and the Proxy relay path are available. A Connector request that cannot obtain an Authority decision is denied and is not forwarded. |
| Stop condition | Stop if a supposedly tunnel-only Connector can still reach a direct Authority path, or if the tunnel is down and new sessions are being treated as allowed. |
| Recovery | Restore the prior firewall rule, inspect relay and reconnect metrics, and verify the configured route before retrying. |
| Rollback | Restore the prior authority_transport and firewall rules together. |
| Escalation | Network owner for routing, Proxy owner for relay, Authority owner for decision availability. |
The reverse tunnel is the data plane. tunneled carries the Authority control
plane inside it; direct and fronted use their configured egress routes. A
Kubernetes request, database statement, model call, web request, or MCP request
can require a per-request Authority decision, so existing data sessions do not
prove a new session is authorized.
AUTHORITY_FRONT="<authority-front>"
AUTHORITY_PORT="<authority-control-port>"
ENROLL_PORT="<authority-enroll-port>"
nc -z -w 3 "$AUTHORITY_FRONT" "$AUTHORITY_PORT"
nc -z -w 3 "$AUTHORITY_FRONT" "$ENROLL_PORT"
The check is intentionally performed from the Connector host. A configured mode that leaves the direct path routable proves nothing.
Authority failover¶
Procedure gate¶
| Gate | Contract |
|---|---|
| Classification | Incident action |
| Prerequisites | Ordered [endpoints].authority_failover addresses, a shared persistent store, Proxy and Connector logs, and a controlled client. |
| Owner | Authority and Proxy incident operators. |
| Approval | Incident authority before changing endpoint order or directing traffic manually. |
| Backup point | Preserve the store, audit export, and current endpoint configuration before intervention. |
| Command or check | Inspect Proxy relay outcomes and run gdsgate --config <operator-config> authority list-nodes --health; then perform a controlled client request through the surviving endpoint. |
| Expected status and exit | A follower refuses the leader-only calls; the Proxy repeats a refused call, or one that reached no instance, at the next endpoint, and the endpoint that answered becomes sticky. A tunnelled Connector's next relay stream reaches the new leader with no client traffic; while no instance holds the lease, the Proxy asks every 250 ms up to every 5 s. A short burst of unavailable calls during leadership movement is expected. |
| Stop condition | Stop admission if no endpoint serves, audit shipping does not recover, or authority verify-audit fails. |
| Recovery | Confirm store lease and leader ownership, find Authority pool moved to another instance in the Proxy log, and watch relay and reconnect counters return to baseline. |
| Rollback | Restore the previous endpoint order only after the failed endpoint is understood. Do not split the store or create a second authority history. |
| Escalation | Database owner for the shared store, Authority owner for leadership, Proxy/network owner for relay reachability. |
The Proxy creates lazy channels and does not hold an active socket to every Authority. A tunneled Connector has no Authority address of its own; the Proxy relays each control-plane stream to the endpoint it finds leading when the stream opens. A long-lived tunnel does not pin the Connector to a stale leader. Every instance serves registration, token-less renewal, workload issuance and administrative commands (calls by instance); the Proxy relays a renewal to the next endpoint when one refuses the connection.
While no instance holds the lease, the audit rows of those calls wait in
audit_outbox, and gdsgate_audit_outbox_oldest_age_secs grows. The new
leader seals them on its first pass. Every instance keeps watching the queued
rows it has seen through the failover, and reports one that leaves the queue
without reaching the chain (queued rows). A row removed
while no instance runs is not reported.
The surviving instances decide by the store's active policy version within
10 s and hold the stored CA state within 5 s of a change, including one
committed by the failed instance just before it stopped. Check
gdsgate_policy_version_applied against gdsgate_policy_version_stored and
gdsgate_ca_bundle_stale on each instance
(policy and CA reload).
A CA rotation the failed leader started is resumed by the new leader from the stored phase, after the rest of the stored propagation or retire wait. A write the former leader makes after the takeover is refused by the store.
Backup and disaster recovery¶
Procedure gate¶
| Gate | Contract |
|---|---|
| Classification | Planned change |
| Prerequisites | An approved backup destination, store credentials supplied outside committed files, external audit keys, node state directories, and a restore test window. |
| Owner | Database, Authority, and security custody owners. |
| Approval | Backup and retention approval before changing the schedule or destination. |
| Backup point | Take a paired store and key backup before every migration, restore, or destructive maintenance. |
| Command or check | Use the backend-native backup command, such as pg_dump --format=custom --file "$BACKUP_FILE" "$STORE_URL" for PostgreSQL, or copy a quiesced SQLite store. Run sha256sum "$BACKUP_FILE", restore into an isolated store, and run gdsgate --config <restore-config> authority verify-audit. |
| Expected status and exit | The backup command and checksum exit 0; the restore copy opens and the audit verifier exits 0 with its external key. |
| Stop condition | Stop if a key is missing, a backup is unverified, the store is in-memory, or a restore cannot verify the chain. |
| Recovery | Retain the failed backup for investigation, make a new backup, and do not delete the last known-good set. |
| Rollback | Restore the prior store, keys, and node state as a paired set. |
| Escalation | Security for key custody, database owner for store integrity, Authority owner for service recovery. |
Back up Authority's state store, which contains the transport CA, SSH CAs,
registry, catalog, request state, and audit chain. Back up both keyrings of
[security.integrity] and
their custody records, each separately from the store, and any key file the
deprecated [audit].*_key_path parameters still name. Keep
anchor checkpoints and exported JSON lines off the Authority host. Persist each
node's state_dir, including Connector SSH host-key material. A database-only
backup cannot prove the same anchored history after restore.
Restoring a cluster¶
Procedure gate¶
| Gate | Contract |
|---|---|
| Classification | Incident action |
| Prerequisites | An isolated restore host, approved store and key backups, the matching verified binary, closed traffic, and an operator with Authority recovery authority. |
| Owner | Authority incident commander with database and security owners. |
| Approval | Incident authority before restoring or opening traffic. |
| Backup point | Preserve the failed cluster and every candidate backup before restore. |
| Command or check | Restore the store and external files, run gdsgate --config <authority-config> authority migrate --check, run gdsgate --config <authority-config> authority verify-audit, start Authority, Proxy, and Connectors with their persisted state, then run HTTP readiness and a controlled session check. |
| Expected status and exit | Migration check is current, verification exits 0, /healthz is 200, /readyz becomes 200 only after each role sets readiness, node health and catalog match the backup, and a controlled session opens and closes. |
| Stop condition | Stop before opening traffic on any migration, chain, key, readiness, policy, catalog, or controlled-session failure. |
| Recovery | Keep traffic closed, return to the isolated host, select the last trusted paired backup, and repeat verification. |
| Rollback | Stop restored services and restore the previous paired store, external keys, node state, and verified binary. Do not downgrade migration rows manually. |
| Escalation | Incident commander, security for chain/key mismatch, database owner for restore failure, service owner for readiness. |
Restore the store and external audit material at their configured paths. The Authority resumes its transport CA from the store, and clients with persisted trust anchors can continue to verify existing node identities. Start Proxy and Connectors with their persisted state directories. Clients reuse cached identity tokens until expiry and then sign in again. Treat migrations as forward-only: an older binary refuses a store carrying versions it does not know.