Deploy Authority in HA¶

The deployment administrator runs several Authorities on a shared PostgreSQL store. One audit write leader holds a fenced lease. Other ready instances serve registration and forward leader-dependent requests through individual private peer addresses. HA covers control-plane failover; it does not increase write capacity or preserve Proxy streams.

Use the same Authority group with one Proxy or an HA Proxy pool. Callers can use individual Authority addresses or an Authority L4 LB. The admission and keyring procedure is the same in both cases.

Before you start¶

Input	Required value / check	Reference
Store	One shared PostgreSQL database reachable from every Authority	State store
Instance identity	Generated stable `ha.owner`; first setup seeds its individual peer	HA fields
Peer routing	Individual HTTPS `50051` address per Authority, reachable by every other Authority	Listener ports
Registration TLS	Verified HTTPS `50050`; bootstrap verifies `enroll.gdsgate.internal` and a trusted CA fingerprint	Bootstrap
Shared settings	Same OIDC, security profile, policy and CA state	HA reload
Gateway advertisement	Selected Proxy public and tunnel addresses and Connector route on every Authority	Node advertisements
Recovery storage	Per-instance protected root and revision cache, backed up with the database	Recovery

Examples use A 192.0.2.10, B 192.0.2.11, store 192.0.2.30 and optional Authority LB 192.0.2.20. Keep control, registration and peer paths private. For a registration certificate supplied by the operator, verify that its SAN includes enroll.gdsgate.internal; a public Proxy certificate is a separate input. The templates use cluster-issued registration certificates.

Prepare and admit the instances¶

Run bootstrap and synchronization: initialize A, admit B with join --role authority, then verify restart without a new permission. Replace advertised Proxy addresses with the selected gateway entry. Keep the individual Authority addresses independent of any LB address.

The first generated configuration has this shape; the actual owner is generated:

[ha]
enabled = true
owner = "authority-a"

[ha.authority_peers]
authority-a = "https://192.0.2.10:50051"

[ha.keyring_sync]
enabled = true
initialize = true
state_dir = "/var/lib/gdsgate/authority/keyring-sync"
poll_secs = 5

A joining instance uses its own owner and initialize = false. After admission, the authenticated snapshot supplies the current peer map and keyrings. Policy, catalog and CA changes use the shared-store reload paths. The protected role join delivers the shared database connection and OIDC settings. Local provider declarations and credentials, per-host paths and external public certificates remain deployment inputs.

Choose caller routing¶

These settings belong to each Proxy configuration. They do not change the Authority's local bind addresses or its individual peer map. Preserve Proxy listeners and enrollment identity state when editing existing tables.

Individual destinations without an Authority LB¶

Role join enables [proxy].authority_topology. The authenticated membership supplies individual control and registration destinations and refreshes them as members join or leave. No per-Proxy peer list is maintained for new installations. Existing explicit primary and fallback addresses remain overrides.

Every registration destination must present the accepted cluster trust and verification name. A registration request is retried on another address only before dispatch; an uncertain state-changing RPC is not replayed.

Stop an Authority, open a new allowed connection and join a disposable node through the public entry. Inspect control and registration routes separately if one succeeds and the other fails. Static overrides remain available in complete Proxy configuration examples.

Destinations through an Authority L4 LB¶

The LB administrator forwards raw TCP 50050 and 50051 to the same ports on ready Authority instances. Use no TLS termination and no PROXY headers. Include ready followers: they forward leader-dependent calls internally.

On each Proxy, set explicit preferred destinations before startup or restart:

gdsgate setup change --proxy-config /etc/gdsgate/proxy.toml \
  --set proxy:endpoints.authority=192.0.2.20:50051 \
  --set proxy:endpoints.authority_enroll=192.0.2.20:50050

Keep automatic membership enabled. The LB addresses take priority; authenticated individual destinations remain available for failover. Authority peer forwarding continues to use individual addresses. Node enrollment through the public entry keeps its verified registration and renewal route; no manual CA copy is needed.

LB contract	Setting	Verification
Registration	`192.0.2.20:50050` → A/B `50050`	New Proxy enrollment succeeds through the LB
Control	`192.0.2.20:50051` → A/B `50051`	Administrative and client requests recover when either Authority stops
Health	HTTP `GET /readyz` on each Authority's private health listener	Only HTTP 200 `ready` enters the pool
Failure handling	Remove unready backends and close their failed connections	New connections select a surviving backend
Peers	A/B retain direct `https://<node>:50051` addresses	A follower can reach the current leader without the LB

Authority /readyz checks the store on each request. HA also requires a live leader lease; the leader's in-process fencing token must match it. A routed follower must reach the leader. Store, lease or route failure returns HTTP 503; recovery restores readiness without a process restart. A healthy follower stays in the pool. Dependency checks have a two-second deadline; /healthz reports process liveness independently.

For an external checker, replace the templates' loopback health bind with 192.0.2.10:9090 on A and 192.0.2.11:9090 on B. Permit only the health checker and private monitoring. Do not expose /metrics to the public network.

Join new Authorities through the verified public Proxy entry. A separate registration endpoint is an explicit override and must meet the same trust and readiness requirements. Compose listener files: Authority LB.

Apply the hosting platform¶

The HA settings and gdsgate subcommands stay the same. The platform supplies process startup, configuration, secret input and private reachability.

Platform	Run and provision	Persistent state / readiness
Linux systemd	`gdsgate --config /etc/gdsgate/authority.toml service install authority`; `systemctl start gdsgate-authority`	Each role's private directory; shared parent rights; `curl --fail http://127.0.0.1:9090/readyz`
Compose	`docker compose exec -T authority-a gdsgate --config /etc/gdsgate/authority-a.toml authority keys status`; run first-boot commands in a one-shot container with the same mounts	Separate Authority volumes, one shared store; Authority LB topology, Compose tab
Kubernetes	Run the gdsgate commands in the named Authority pod or a controlled Job with the same configuration and state	Distinct durable per-instance volumes and stable owners; individual pod addresses for peers, optional Service for callers; platform deployment
Swarm	Run each configured Authority with its own stable owner and mounted protected state	Individual task destinations for peers; a service VIP can be a caller destination; platform deployment

Mount bootstrap permission files as private readable inputs, not arguments. Start the first Authority before joining another. Preserve per-instance state across process recreation; a new empty volume is a new bootstrap operation, not an ordinary restart. Restrict cross-node paths even when they use TLS.

Verify and recover¶

The operator runs these checks after bootstrap, with callers configured using the chosen route. Restore each stopped instance before the next failure test.

Check	Expected	Recovery
`authority keys status` on every instance	Same revision and expected members	Restore store access; read the affected Authority's logs
`doctor` with each Authority configuration	No failed trust, store or state checks	Fix the named check before publishing readiness
`/readyz` on every instance	`ready`	Inspect store, TLS, protected state and leader forwarding
Stop the current write leader, then open an allowed resource through Proxy	New access recovers after lease takeover; default lease TTL is 15 seconds	Inspect the surviving node's lease, peer path and store connectivity
Stop a primary registration destination, then enroll a disposable Proxy	Registration succeeds via the fallback or ready LB backend	Inspect the registration route and certificate verification name
Public workstation tries Authority `50050`/`50051`	Network refusal	Correct firewall or LB exposure

Registration and control use different listeners: test both. A successful ls alone does not establish recovery of leader-dependent access decisions. Existing sessions follow the Proxy failure behavior.

Use rotation, adding an Authority and paired recovery for later changes. Production rotation preparation requires a declared generation_kms_key; this is separate from native bootstrap delivery. After component verification, continue with Proxy HA or deployment acceptance.