Bootstrap and synchronize Authority¶

The deployment administrator initializes the first HA Authority and admits additional instances with expiring, owner-bound bootstrap permissions. Keyring changes then reach every admitted Authority automatically.

Use this procedure with either individual Authority addresses or an Authority L4 load balancer, and with one or several Proxies. First select the Authority routing and platform; the composed deployments show both routing topologies with Linux and Compose instructions. Bootstrap depends on Authority registration and the shared store, not on the placement or number of Proxies.

The commands below use Linux with systemd and the pilot profile. For containers, keep the same gdsgate commands and use the platform mapping. Commands run in a root shell.

Before you start¶

Item	Check / value	Reference
First Authority A and joining Authority B	Private addresses `192.0.2.10` and `192.0.2.11`; stable owners `authority-a` and `authority-b`	Authority HA
gdsgate and the selected service manager on both hosts	`gdsgate --version`; Linux examples use `systemctl --version`	Installation
New deployment directories and empty cluster database	For an existing cluster use migration	State store
Shared PostgreSQL database, non-superuser application role and protected password file	`psql postgres://gdsgate@192.0.2.30:5432/gdsgate -W -c 'select 1'` from both hosts	State store
Same OIDC issuer, client and security profile	Example issuer `https://idp.example.com/realms/gdsgate`, client `gdsgate`, group `admins`	OIDC
Protected transfer and synchronized clocks	Transfer account `ops`, home `/home/ops` on B; `timedatectl status`	Secret references
Gateway addresses supplied by the Proxy deployment owner	Templates advertise `203.0.113.10:50061`–`50064`, route `tunneled`; replace with the selected Proxy entry addresses	Node advertisements

Open the Authority paths¶

The network administrator permits these private paths before bootstrap:

Listener	Allow from	Purpose
A `50050`	Joining Authorities and authorized enrolling nodes	Pinned TLS bootstrap and registration
Every Authority `50051`	Other Authorities and authorized control-plane callers	Mutual TLS control and leader forwarding
Store `5432`	Every Authority	Shared state and encrypted keyring snapshots
Each Authority health listener	Local operator; Authority LB health checker when used	`/readyz` and metrics; loopback `9090` in the templates

Use Authority routing to configure callers and an optional LB. Public Proxy listeners and Proxy peering are configured in Proxy HA.

Check the store from both Authority hosts before initializing A:

psql postgres://gdsgate@192.0.2.30:5432/gdsgate -W -c 'select 1'

From an untrusted workstation, TCP access to 192.0.2.30:5432 must fail. Use a private database listener and allow only Authority addresses. A container store should have no published host port when every Authority reaches its private network. If remote Authorities need a published port, verify the container forwarding rules as well as the host input firewall. A host firewall rule alone does not establish isolation of a published container port.

The application role must own its database and have no superuser or role administration privileges. PostgreSQL and its backup availability are separate from the Authority process pool.

Configure¶

Run these steps in order. Keep callers out of the pool until both Authorities are ready.

1. Download the configurations¶

Both hosts: download the templates into /opt/gdsgate-bootstrap:

install -d -m 0750 /opt/gdsgate-bootstrap
cd /opt/gdsgate-bootstrap
for file in authority-a.toml authority-b.toml; do
  curl -fsS "https://gate.gds.dev/docs/ha/files/authority/$file" -o "$file"
done
groupadd --system --force gdsgate
install -d -m 0750 -g gdsgate /etc/gdsgate

Complete configuration	Install on	Purpose
authority-a.toml	A	Initialize both rings; publish both individual Authority addresses
authority-b.toml	B	Bootstrap from A; no local ring files or initial peer map

Edit the private addresses, advertised gateway addresses and OIDC settings in these files. --domain in step 3 must name the selected gateway entry. These are complete configurations; install each one as a file. Keep one instance of each TOML table.

2. Install the database password¶

Both hosts: install the same database password from a protected input file named store.password. It contains only the password:

install -o root -g gdsgate -m 0640 store.password /etc/gdsgate/store.password

The root-owned file is readable by service accounts through group gdsgate. Keep the password out of the database URL and shell arguments. Remove the transfer copy once the database connection check succeeds.

3. Prepare the first Authority¶

Host A: create the initial rings, seed policy and cluster-issued registration certificate. Run with --dry-run first:

gdsgate setup prepare --domain 203.0.113.10 --security-profile pilot \
  --role authority --authority-bind 192.0.2.10 --join-relay \
  --authority-output /etc/gdsgate/authority.toml \
  --policy-output /etc/gdsgate/policy.cedar \
  --store-url postgres://gdsgate@192.0.2.30:5432/gdsgate \
  --store-password-file /etc/gdsgate/store.password \
  --oidc-issuer https://idp.example.com/realms/gdsgate --oidc-client-id gdsgate \
  --access-group admins --dry-run

Resolve any x: refusal, then run the same command without --dry-run. Do not add --start: the HA configuration must be installed first.

Expected files:

File	Use
`/etc/gdsgate/control.keys`, `/etc/gdsgate/audit.keys`	Initial integrity rings, generated only on A
`/etc/gdsgate/policy.cedar`	Initial policy for group `admins`
`/var/lib/gdsgate/authority/enroll.pem`, `enroll.key`	Registration certificate and key, renewed by Authority
`/etc/gdsgate/transport-ca.pem`	Public CA certificate; includes no private key

Install A's HA configuration and service:

install -m 0640 -o root -g gdsgate /opt/gdsgate-bootstrap/authority-a.toml /etc/gdsgate/authority.toml
gdsgate --config /etc/gdsgate/authority.toml service install authority

A has ha.owner = "authority-a", both private peers and ha.keyring_sync.initialize = true. Its first start creates the encrypted canonical snapshot from the generated rings.

4. Prepare the joining Authority¶

Host A: read the CA fingerprint and transfer only the public CA to B:

openssl x509 -in /etc/gdsgate/transport-ca.pem -noout -fingerprint -sha256
scp /etc/gdsgate/transport-ca.pem ops@192.0.2.11:transport-ca.pem

Record the displayed fingerprint through your trusted operator channel. Do not copy A's control.keys, audit.keys, registration private key or distribution root to B.

Host B: install the CA and configuration:

install -m 0644 -o root -g gdsgate /home/ops/transport-ca.pem /etc/gdsgate/transport-ca.pem
rm /home/ops/transport-ca.pem
install -m 0640 -o root -g gdsgate /opt/gdsgate-bootstrap/authority-b.toml /etc/gdsgate/authority.toml
openssl x509 -in /etc/gdsgate/transport-ca.pem -noout -fingerprint -sha256

Compare that fingerprint with A's value before continuing. Replace REPLACE_WITH_TRANSPORT_CA_FINGERPRINT in B's configuration with the SHA-256 hex value, without colons. B has ha.owner = "authority-b", initialize = false, and bootstrap endpoint https://192.0.2.10:50050. B's own registration certificate is issued after bootstrap from the shared cluster CA; it needs no manually generated TLS pair.

Start¶

5. Start the first Authority¶

Host A: start Authority, check readiness and take the administrative identity used for subsequent administration and node enrollment:

systemctl start gdsgate-authority
curl --fail --silent http://127.0.0.1:9090/readyz
gdsgate --config /etc/gdsgate/authority.toml authority keys status
gdsgate --config /etc/gdsgate/authority.toml authority bootstrap-admin \
  --output-dir /etc/gdsgate/admin

Before starting, service install authority repairs the shared /var/lib/gdsgate parent to 0755 and prepares the private role directory. For a working-directory refusal, inspect namei -l /var/lib/gdsgate/authority and reinstall the service with this configuration. Do not make the role's private directory or /var/lib/private public to bypass the refusal.

Expected: ready; key status includes distribution revision 1 · 2 mandatory Authority members. Stop here if A is not ready. Diagnose with journalctl -u gdsgate-authority -n 50.

6. Bootstrap the second Authority¶

Host A: issue B's bootstrap permission directly into a private file:

umask 077
gdsgate --config /etc/gdsgate/authority.toml authority keys sync create-token \
  --owner authority-b --ttl-secs 600 > /etc/gdsgate/authority-bootstrap.token
scp -p /etc/gdsgate/authority-bootstrap.token ops@192.0.2.11:authority-bootstrap.token
rm /etc/gdsgate/authority-bootstrap.token

The permission expires after 600 seconds and is bound to authority-b. Command options: bootstrap permissions. scp -p preserves the transfer file's 0600 mode. Do not print its contents, put them in arguments, or reuse it for a different owner.

Host B: install the permission, install the service and start it:

install -m 0640 -o root -g gdsgate /home/ops/authority-bootstrap.token /etc/gdsgate/authority-bootstrap.token
rm /home/ops/authority-bootstrap.token
gdsgate --config /etc/gdsgate/authority.toml service install authority
systemctl start gdsgate-authority
curl --fail --silent http://127.0.0.1:9090/readyz
gdsgate --config /etc/gdsgate/authority.toml authority keys status

Expected: ready and the same distribution revision as A. Under /var/lib/gdsgate/authority/keyring-sync, B now holds root.key, receiver.key and snapshot.enc, owned by its Authority service account. Bootstrap checks pinned TLS and completes before readiness.

7. Verify restart without a bootstrap permission¶

Host B: after those checks pass, remove the one-time input and its three configuration fields, then verify a restart:

rm /etc/gdsgate/authority-bootstrap.token
sed -i '/^bootstrap_endpoint =/d; /^bootstrap_fingerprint =/d; /^bootstrap_token =/d' /etc/gdsgate/authority.toml
systemctl restart gdsgate-authority
curl --fail --silent http://127.0.0.1:9090/readyz

Expected: ready without a bootstrap token. On A, also verify a live store-backed control-plane read using its administrative configuration:

gdsgate --config /etc/gdsgate/authority.toml authority list-nodes

Authority readiness checks the shared store and live leader lease. With internal peer routing, a follower also checks its authenticated route to the leader. After Proxy enrollment, verify policy pull through a logged-in client and a new native resource connection. Keep root.key, receiver.key and snapshot.enc; subsequent starts authenticate the current snapshot using that protected state.

Verify¶

The deployment owner checks each Authority before admitting callers.

Host / command	Expected	Recovery
Both: `gdsgate config check /etc/gdsgate/authority.toml --role authority`	No errors after bootstrap	Inspect the named file and service ownership
Both: `curl --fail --silent http://127.0.0.1:9090/readyz`	`ready`	Read `journalctl -u gdsgate-authority -n 50`; inspect store and TLS errors
Both: `gdsgate --config /etc/gdsgate/authority.toml authority keys status`	Same distribution revision and both rings; both owners acknowledge the revision after polling	Restore the missing node's service and store connection
B: restart after removing the bootstrap input	`ready`; protected root and revision cache remain	Restore paired protected state; do not request a new identity by deleting it
B, before its first successful admission: wrong fingerprint with fresh joining state	Bootstrap refused before readiness	Restore the fingerprint obtained through the trusted channel; never disable TLS verification

Run the negative bootstrap check before admitting B; do not change a provisioned node's protected state or add a member only for this test. After these checks, complete Authority failover verification, then deploy Proxy and Connector, or use the existing callers.

Rotate synchronized keyrings¶

Run on either provisioned Authority as root. Both Authority services must be running, share the current revision and have synchronized clocks. Keyrings hold the material that seals control rows and audit records.

gdsgate --config /etc/gdsgate/authority.toml authority keys rotate control --prepare
gdsgate --config /etc/gdsgate/authority.toml authority keys status

Expected: a successor distribution revision and one prepared control key. Check keys status on both hosts after at least one poll_secs interval (default 5 seconds). Both must hold the prepared key before activation:

gdsgate --config /etc/gdsgate/authority.toml authority keys rotate control --activate

Expected: a new revision; the prepared key becomes active and the former active key remains for verification. Replicas reload without file copying or service restarts. Activation requires an acknowledgement of the exact prepared revision, no older than 30 seconds, from every mandatory owner.

Negative check in an acceptance window: stop B after prepare and wait more than 30 seconds. Activation on A must fail without changing the canonical revision. Start B, wait for it to load and acknowledge the revision, then repeat activation. Do not remove B from membership to bypass the check.

Task	Command / constraint	Reference
Audit HMAC rotation	`authority keys rotate audit --prepare`, then `--activate`	Keyring commands
Checkpoint signing rotation	`authority keys rotate audit --prepare --kind signing`, then `--activate`; prepare one kind at a time	Keyring commands
Control retirement	`authority keys rotate control --retire`; existing sealed-row and key-loss guards apply	Keyring commands
Audit retirement	Refused: historical audit keys remain available	Audit verification

A local edit of the original control.keys or audit.keys does not publish a new revision while synchronization is enabled.

Rotate the distribution root¶

Run as the local administrator on a provisioned Authority. Update every Authority to a build with root rotation support before preparing a transition. Each member must have its own receiver.key bound to its stable owner.

Root rotation replaces the credential that encrypts canonical integrity snapshots. It preserves existing control and audit material. To replace those keys, use keyring rotation after the root transition completes.

On the initiating Authority:

gdsgate --config /etc/gdsgate/authority.toml authority keys sync prepare-root
gdsgate --config /etc/gdsgate/authority.toml authority keys sync root-status

Expected: a preparation ID, the initiator, and an acknowledgement state for every retained owner. Wait until every owner reports acknowledged, then activate on the same Authority:

gdsgate --config /etc/gdsgate/authority.toml authority keys sync activate-root
gdsgate --config /etc/gdsgate/authority.toml authority keys sync root-status
curl --fail --silent http://127.0.0.1:9090/readyz

Check root-status and /readyz on each retained host after polling and lease handoff. Expected: the next root epoch, no pending preparation, and ready. Replicas reload without restarting. During local installation, readiness and new RPC admission are blocked; leadership resumes after successful verification. Ordinary ring edits, membership additions and bootstrap issuance are blocked while a preparation is pending.

Withdraw an Authority¶

Retain at least two owners. To withdraw from a two-node cluster, first bootstrap a replacement Authority and confirm its readiness. Prepare on an owner that will remain in the cluster:

gdsgate --config /etc/gdsgate/authority.toml authority keys sync prepare-root \
  --withdraw authority-c
gdsgate --config /etc/gdsgate/authority.toml authority keys sync root-status
gdsgate --config /etc/gdsgate/authority.toml authority keys sync activate-root

Repeat --withdraw for each explicitly reviewed removal. Only retained owners receive the new root and contribute required ACKs. A missing retained owner blocks activation; a timeout never withdraws it. After commit, a removed Authority exits and refuses restart with its former membership. Disable its service and remove its addresses from caller failover lists and any Authority LB.

Withdrawal prevents future root delivery to that owner. It does not revoke previously known CA, control or audit keys, or separate DB/KMS credentials. If the host was compromised, rotate or revoke those credentials through their respective procedures.

Cancel or recover a preparation¶

If preparation cannot finish, run on any retained Authority:

gdsgate --config /etc/gdsgate/authority.toml authority keys sync cancel-root
gdsgate --config /etc/gdsgate/authority.toml authority keys sync root-status

Expected: the same root epoch and no pending preparation. Cancellation advances the material revision, invalidates the old preparation and lets replicas remove its local journal. If the initiator is lost, cancel from a retained owner and prepare again there. Cancellation and activation are serialized; only one can commit. After an interrupted administrative command, inspect root-status before retrying.

Retained Authorities recover an interrupted committed root/cache write on restart or the next poll. An old paired backup can catch up through committed generations using its independent receiver credential. Keep the entire keyring-sync directory together, including root.key or root.enc, receiver.key, snapshot.enc and any root-install.json. A wrapped root also requires its original wrapping provider. Do not delete the cache watermark to accept a rolled-back database.

Diagnose and recover bootstrap¶

Run diagnostics on the node whose service failed. Read its log before changing protected state:

journalctl -u gdsgate-authority -n 50
systemctl status gdsgate-authority

Symptom	Cause	Recovery
TLS chain or pin refusal	Wrong CA fingerprint, unreachable A, or missing issuer chain	Check A's `50050`, registration chain and the fingerprint recorded in step 4; do not disable TLS checks
`bootstrap request does not match this permission`	A new permission was supplied with the previous persisted request	Follow the expired-permission procedure below while the joining service is stopped
Bootstrap permission denied	Wrong owner, invalid/expired grant, or reuse with another request	Keep `ha.owner = "authority-b"`; issue a new permission if expired
Local storage permission denied	The service cannot traverse/write its state directory or read its input	Inspect `namei -l`; rerun `service install authority` with this configuration; retain owner-only protected state
`invalid or unauthenticated keyring distribution state`	Tamper, wrong root, rollback or divergent material at the same revision	Stop the affected instance and restore paired protected state; do not delete its watermark to accept an older database
Activation refuses	Missing, stale or different-revision member acknowledgement	Restore that member's database access and service, then wait for polling and retry

If A committed the bootstrap but its response was lost, B's next start uses the persisted request nonce to recover the same response while the permission remains valid. Do not remove that request for an ordinary connection retry.

If the permission expired before B persisted root.key, Host B:

systemctl stop gdsgate-authority
test ! -e /var/lib/gdsgate/authority/keyring-sync/root.key && \
  rm -f /var/lib/gdsgate/authority/keyring-sync/bootstrap-request.json

Keep snapshot.enc if it exists. Obtain and install a new permission using step 6, restore the three bootstrap fields from B's template with the trusted fingerprint, then start B. If root.key already exists, recover its protected state from the paired backup.

An unavailable or emptied store can surface as Authority leader discovery failed in a client. Check the database and leader lease before changing Proxy TLS or issuing more enrollment tokens. Withdraw affected services until live policy reads and new resource access succeed.

Recreating an empty store creates a new cluster unless paired state is restored. Its CA fingerprint, administrative identities, node grants and catalogue must be established again. Existing client anchors and enrolled nodes do not become members of that new cluster by restarting them. Choose paired recovery or new provisioning explicitly; use join trust checks when onboarding to a newly provisioned cluster.

Back up the shared database and each Authority's protected root (root.key, or root.enc with a wrapping provider) with receiver.key and snapshot.enc as a paired recovery set. The database alone contains no usable distribution root. A restored older database cannot replace a newer local revision watermark. Recovery procedure: Backup and disaster recovery.

Add another Authority¶

Prepare the new host as B, using its own private address and owner. On an already provisioned Authority, create its permission with the new control address:

umask 077
gdsgate --config /etc/gdsgate/authority.toml authority keys sync create-token \
  --owner authority-c --peer https://192.0.2.12:50051 > /etc/gdsgate/authority-c-bootstrap.token

The command publishes the new peer in the synchronized membership and issues an expiring permission. The new owner becomes a mandatory rotation participant immediately. Deliver the permission and perform steps 6–7 with ha.owner = "authority-c". Publish the new ready node to callers using the selected Authority routing: update individual fallback lists or the Authority LB pool. Caller configuration changes require their normal reload or restart; Authority membership reloads automatically.

Check the same revision and key holders on the new host before any rotation. This flow does not implement removal or revocation of a provisioned Authority.

Choose the bootstrap endpoint¶

First-Authority initialization, owner-bound permissions and keyring rotation are identical with or without an Authority LB. For first boot, point B's bootstrap_endpoint directly at provisioned A's HTTPS registration listener. Keep each Authority's individual control address in the peer map.

An Authority registration LB endpoint is also valid if every healthy backend is already provisioned and presents the pinned registration CA and expected TLS name. Include only ready instances in that pool. Check membership before joining a node; a public Proxy LB is a separate gateway entry point.

Ordinary control and registration destinations for Proxies are selected in Authority HA, independently of this bootstrap endpoint.

Migrate an existing cluster¶

Do not run the new-deployment setup prepare command over an existing cluster. Take a protected recovery backup and schedule the restart of its Authorities.

On the existing first Authority, retain the current store, OIDC, policy, listener and certificate settings. If keys still use legacy per-domain files, run authority keys migrate-legacy before enabling sync. Add the [ha], [ha.authority_peers] and [ha.keyring_sync] values from A's template using the real owners and private addresses. Keep both existing integrity ring references as the initializer's seed.

Start the first Authority and verify its canonical revision. Configure each other Authority with initialize = false and its own owner, then apply the joining and verification steps above. Local edits of old ring files are no longer a distribution mechanism. Keep the paired backup until every instance has passed restart and keyring checks. Rollback requires restoring consistent database and integrity material; switching off sync while reusing stale seed files is not a safe rollback.

Profile and limits¶

Setting / boundary	This procedure	Next action
Security profile	`pilot`, IP-only TLS issued by the cluster	Profile checks
`production` key generation	Rotation prepare needs `ha.keyring_sync.generation_kms_key` naming a declared provider	Keyring sync reference
Automatic synchronization	Integrity rings and Authority peer membership	Policy, catalogue and CA state use their existing shared-store reloads
Operator-owned inputs	Database/OIDC/provider credentials, node config and external TLS files	Deliver through deployment tooling; they are not copied by bootstrap
Trust after admission	Every provisioned Authority holds both rings and the distribution root	Removing a peer address does not revoke this trust
Unsupported lifecycle	Distribution-root rotation and admitted Authority removal/revocation	Preserve the current trusted membership; use a separate planned lifecycle procedure
Service commands	Linux systemd examples	Use the platform mapping for containers

Field syntax, defaults and ownership: [ha.keyring_sync]. After verification, continue with Authority verification and the selected Proxy deployment.