High availability configuration

Paths in the tables are relative to [ha]. Check parsing and cross-field rules with gdsgate doctor --config <PATH> --json. See the Authority HA procedure.

[ha]

Used by: Authority; max_cluster_sessions also applies to Proxy.

Several Authority instances on one shared PostgreSQL store_url, with one audit write leader at a time. Off by default. Each takeover increments a fencing token on the lease row; a former leader's audit commit and CA rotation write are refused once another instance has taken the lease.

Key	Type	Default	Purpose
enabled	bool	false	Turns HA on.
owner	string	""	This instance's lease owner id. Empty: a per-process id is generated. Set it to see a stable leader name.
lease_ttl_secs	u64	15	Lease validity. A follower takes over this long after the leader stops renewing.
renew_secs	u64	5	Lease renewal interval of the leader and acquisition retry interval of followers. Keep it well below lease_ttl_secs.
authority_peers	map of strings	{}	Stable lease owner to individual HTTPS control-plane address. Enables internal forwarding; requires HA and an explicit owner present in the map. Include every Authority.
max_cluster_sessions	u64	1024	Shared live-session UID limit. 0 selects 1024; maximum 4096. Set the same value on all Proxies. Local Proxy and peer limits also apply.

[ha]
enabled        = true
owner          = "auth-eu-1"
lease_ttl_secs = 15
renew_secs     = 5
max_cluster_sessions = 1024

[ha.authority_peers]
auth-eu-1 = "https://authority-a.example.com:50051"
auth-eu-2 = "https://authority-b.example.com:50051"

HA provides failover only; writes do not scale.

Closing a session releases its shared capacity. If the close races a leader change or a temporary outage, the Proxy retries cleanup within the existing lease plus one final RPC deadline. Cleanup does not renew the lease; an unreachable Authority leaves the record to expire.

Calls	Served by
Node registration and renewal; workload identity issuance and renewal; delegation renewal; administrative commands on every route (gateway, certificate, emergency)	any instance
Access decisions and certificates for sessions, audit shipping and recordings from connectors, step-up and MFA, access requests, delegation creation and revocation	the leader; with authority_peers, a follower forwards the request internally
CA rotation phases, lease expiry, sealing queued audit rows	the leader, in the background

A follower queues the audit rows of the calls it serves. The leader seals waiting rows every renew_secs, after each write it makes, and on each lease sweep. renew_secs sets how often the leader looks. How long a row waits has no upper bound: with no leader, or with more rows than one pass takes, rows wait until an instance holds the lease and reaches them. gdsgate_audit_outbox_oldest_age_secs shows how long the oldest has waited.

Every instance loads the same audit keyring. A queued row is sealed under a key derived from its anchor key, the leader holds a row whose seal does not verify, and every instance reports a queued row it has seen that leaves the queue before an anchored chain row carries its event or the leader's hold of it. See queued rows.

Configuration	Routing
authority_peers populated	A ready follower reaches the current leader through its individual address. The Proxy uses its configured individual Authority addresses or a common LB address.
authority_peers empty	The Proxy uses endpoints.authority and authority_failover; definite leader refusals select another instance.

With an Authority TCP LB, expose every ready Authority through it. Without one, configure both private addresses on every Proxy. Keep the peer map identical on all Authorities; each address must reach one instance directly. The leader checks the forwarding Authority's enrolled identity and current lease token. Ambiguous failures return to the caller without replay. Configuration and checks: Authority routing.

Policy and CA reload

Every instance decides by the store's active policy version and holds the stored CA state. Each runs two reload loops; the first pass runs at start.

Loop	Every	Each pass
Policy	10 s	Compares the version this instance decides by with the store's active version, and loads the active one when they differ.
CA	5 s	Compares each CA kind's stored generations, trust bundle and rotation phase with the state this instance last loaded, and reloads a kind that differs.

A change committed on any instance is in force on every instance within one interval. A load that fails leaves the instance on the policy or CA it holds; the next pass tries again, with no further change needed.

A policy push, vote, rejection and break-glass restore each take a lock on the policy in the store for their transaction. Instances apply them one at a time, on the state the previous one left: a proposal is activated or rejected, a vote is never lost, and of two changes on one base version one lands and the other is refused as stale. A policy change, its reload notice and its audit row commit together.

A CA rotation request (gdsgate authority ca rotate user-ssh and the other rotate-* commands) stays in the store until a rotation of that CA starts on the write leader. A start that fails, a leader that stops, or a request made while no Authority runs leaves it for the next check on whichever instance leads. Each rotation phase is written in one transaction.

A rotation runs under the lease token its leader held when the rotation started. Each phase write carries that token and the revision of the CA state the leader loaded, and the store checks both inside the write's transaction. A write from an instance that lost the lease, or that holds an older state, is refused and changes nothing. A leader that steps down stops its rotation within a second.

The end of the propagation wait and of the retire wait is stored with the phase. An instance that resumes a rotation, after a restart, on a new leader or after a failed write, does not take the next step before that time. A step whose write failed is dropped from memory, and the next check starts from the stored phase. Bound: the clock difference between instances. An instance whose clock runs ahead of the one that stored the phase leaves it early by that difference.

Symptom	Cause	Action
gdsgate_policy_version_applied below gdsgate_policy_version_stored for more than 20 s	The active version does not load on this instance. The log line could not load the store's active authorization policy carries the error every pass.	Fix the cause in the log. A database error clears when the database recovers. A version that does not parse needs a new push; the instance keeps the previous version until then.
gdsgate_ca_bundle_stale{kind} is 1	The stored CA of kind changed and this instance could not load it. The log line could not load it; keeping the CA it holds names ca.	Fix the cause in the log. The gauge returns to 0 on the pass that loads it.
A requested CA rotation does not start	The leader's log line the CA rotation did not complete names ca and carries the error on every check.	Fix the cause in the log. The rotation starts on the next check; the request does not need repeating.
the CA rotation did not complete with stale audit-write fencing token or the stored state of CA kind ... moved	Another instance took the lease, or wrote the CA state, while this one waited inside the rotation. Its write was refused.	None. The leader resumes the rotation from the stored phase on its next check.

[ha.keyring_sync]

Opt-in canonical integrity material and owner-bound first boot. See the deployment procedure. Used by Authority.

Key	Type	Default	Purpose
enabled	bool	false	Read authenticated encrypted snapshots and reload running signers. Requires HA, explicit owner and TLS registration.
initialize	bool	false	First Authority only: initialize an empty source from both existing integrity rings. An existing local watermark prevents reinitialization after a database restore.
initialize_if_empty	bool	false	Refuse automatic first initialization when the store already contains resources, nodes, policy or audit. New PostgreSQL setup writes true; explicit standalone migration retains the existing material.
state_dir	string	/var/lib/gdsgate/authority/keyring-sync	Owner-only root, independent receiver credentials and encrypted revision cache. Windows: C:\ProgramData\gdsgate\authority\keyring-sync. Back up this directory as one protected recovery set.
bootstrap_endpoint	string or absent	absent	Individual HTTPS registration endpoint of an existing Authority, used only when no local root exists.
bootstrap_fingerprint	string or absent	absent	Pinned registration CA fingerprint, obtained through a trusted channel.
bootstrap_token	secret reference or absent	absent	Short-lived grant bound to this instance's ha.owner. For example ${file:/etc/gdsgate/authority-bootstrap.token}.
generation_kms_key	string or absent	absent	Name from kms_keys that generates rotation material. Required for --prepare in production; optional in other profiles. Independent of bootstrap delivery.
root_kms_key	string or absent	absent	Name from kms_keys that wraps the local distribution root as state_dir/root.enc. Select before provisioning this Authority. Unset uses protected root.key.
poll_secs	u64	5	Reload and acknowledgement interval, from 1 to 30 seconds. Activation requires acknowledgements no older than 30 seconds from every configured member.

Startup validates and persists the current snapshot before readiness. Prepare, activation and membership additions use authenticated compare-and-swap; member receipts do not depend on audit write leadership. Every explicit member must confirm prepare before activation; an offline member is never silently dropped. authority keys status includes the material revision.

With synchronization enabled, the initial peer map is replaced by the snapshot's map and reloaded at runtime. Legacy ring references seed the first Authority only. Local edits of those files do not change the canonical rings. CA material, policy and catalog use their existing store-based reload loops.

Each Authority keeps its independent encryption and signing keys in state_dir/receiver.key. Bootstrap binds their public halves to ha.owner; an existing Authority registers them using its current distribution credential. The binding is immutable. Keep this private file with the root and snapshot cache when backing up or restoring the Authority.

With root_kms_key, startup unwraps root.enc through the selected provider. Provider refusal, an unavailable provider or invalid ciphertext stops startup. Bootstrap still delivers the root through its owner-bound permission, then stores it wrapped. Snapshot reloads use the root held in memory.

For an existing Authority, declare the provider and set root_kms_key, then run on that host before restarting its service:

sudo gdsgate --config /etc/gdsgate/authority.toml authority keys sync root wrap

The command persists and verifies root.enc before removing root.key. An interrupted command can be repeated with the same setting; it retains the same root and does not edit the shared store. A conflicting wrapped root stops migration and preserves the plaintext file.

Changing the setting alone does not migrate the root. An existing root.key with root_kms_key set, or root.enc without the setting, stops startup. Preserve the root, receiver credentials, cache and any root installation journal together in backups; a wrapped-root restore also requires access to the same wrapping provider. Use root rotation and explicit prepare-root --withdraw <OWNER> for membership withdrawal. Editing a caller's peer address list does not revoke a distribution credential.

Failed synchronization blocks new RPC admission, readiness and lease renewal until the root, snapshot and membership are verified. A committed withdrawal stops the removed Authority and refuses startup with its former membership. Bootstrap RPC and the runtime use the same root generation. Root activation requires independently signed durable ACKs from every retained owner; the 30-second acknowledgement window above applies to ordinary keyring activation.