Bootstrap and synchronize Authority¶
Initialize the HA group once. Additional Authorities receive the shared store, cluster settings, keyrings and current topology through an owner-bound encrypted join. Each instance keeps its own receiver credential and private state.
The procedure applies to individual Authority destinations and an Authority L4 LB. It also applies with one Proxy or an HA Proxy pool. Select the Authority routing and platform independently. The complete topologies compose these steps with Proxy and Connector installation.
Before you start¶
| Input | Example / requirement |
|---|---|
| Public Proxy entry | 203.0.113.10, or gate.example.com with its TLS certificate |
| Individual Authority addresses | A 192.0.2.10, B 192.0.2.11; each reachable by the other Authorities |
| Store | Shared PostgreSQL, database and application role gdsgate |
| Identity provider | Issuer https://idp.example.com/realms/gdsgate, client gdsgate, access group admins |
| Local installation | Installed binary, synchronized clocks, persistent private state |
| Trust | Approved CA fingerprint from the initial setup output, or approved public TLS trust |
Use migration for an existing cluster. Initial setup requires a new database and new installation paths. The store and public LB have their own availability requirements.
Open the Authority paths¶
| Listener | Allow from | Purpose |
|---|---|---|
Authority 50050 |
Proxies and authorized registering nodes | TLS registration and protected bootstrap |
Authority 50051 |
Other Authorities and authorized Proxies | Mutual TLS control and leader forwarding |
Store 5432 |
Authorities | Shared state and encrypted snapshots |
Authority health 9090 |
Local operator or private health checker | /readyz and metrics |
The Authority peer addresses remain individual addresses when callers use a private LB. Proxy public ports and its individual peer port are described in Proxy HA. Verify database access from every Authority host before setup. The application database role must have no superuser or role administration privileges.
Initialize the first Authority¶
First installation¶
The example installs Authority and Proxy together on host A. Run in a root shell:
PUBLIC_ENTRY=203.0.113.10
NODE_IP=192.0.2.10
STORE_URL=postgres://gdsgate@192.0.2.30:5432/gdsgate
OIDC_ISSUER=https://idp.example.com/realms/gdsgate
OIDC_CLIENT_ID=gdsgate
gdsgate setup prepare --role pair --domain "$PUBLIC_ENTRY" \
--security-profile pilot --store-url "$STORE_URL" \
--store-password-file /etc/gdsgate/store.password \
--authority-bind "$NODE_IP" --listen "$NODE_IP:50061" \
--oidc-issuer "$OIDC_ISSUER" --oidc-client-id "$OIDC_CLIENT_ID" \
--access-group admins --start
Install the database password at the named protected path before running the
command; it contains only the password. setup prepare reports planned paths and
refuses incomplete profile requirements. Add --dry-run to inspect the plan.
For a DNS public entry, add --tls-cert and --tls-key with the public certificate
and key. IP-only setup uses cluster-issued registration TLS.
A new PostgreSQL Authority enables HA and keyring synchronization immediately.
Its generated owner and individual control address initialize a one-member
topology. The paired Proxy receives an independent transport identity and its
individual peer address. Additional members are discovered after authenticated
admission. No --ha flag or initial list of future nodes is required.
Default Linux configurations are /etc/gdsgate/authority.toml and proxy.toml;
private state is under /var/lib/gdsgate/authority and proxy. Override paths or
ports only when the platform requires it. For Authority on a separate host,
use --role authority, then complete Proxy installation.
Start and check¶
--start installs and starts the Linux or Windows services. Containers omit
--start and use their supervisor with gdsgate authority and gdsgate proxy.
See the platform mapping.
curl --fail --silent http://127.0.0.1:9090/readyz
gdsgate authority keys status
gdsgate authority identities bootstrap --role admin --output-dir /etc/gdsgate/admin
gdsgate setup change --authority-config /etc/gdsgate/authority.toml \
--set authority:admin.identity_dir=/etc/gdsgate/admin
The local administrative identity is separate from user login and from node identities. Administrative commands select the installed Authority configuration. Keep the printed cluster CA fingerprint with the deployment record. On a joined Authority, initialize and select its own local administrative identity before running administrative commands there; role join does not copy the first Authority's administrative key.
Join another Authority¶
On any admitted Authority, create a role permission in a private file:
Transfer only this permission through the protected operator channel. On host B, run in a root shell with the public entry and approved fingerprint:
PUBLIC_ENTRY=203.0.113.10
NODE_IP=192.0.2.11
CA_FINGERPRINT='<approved SHA-256 fingerprint>'
gdsgate join "$PUBLIC_ENTRY" --role authority --listen "$NODE_IP" \
--fingerprint "$CA_FINGERPRINT" --start < authority.token
rm authority.token
The command discovers the protected bootstrap route through Proxy, verifies cluster identity before sending credentials, and assigns this instance its own owner. It installs the shared settings and current membership in a response bound to the receiver. No CA, distribution root, keyring or peer list is copied by hand. Local KMS/provider credentials remain local deployment inputs.
For Authority and Proxy together, issue --role pair and use join --role pair.
For an additional Proxy, use Proxy join. Omitting
--fingerprint permits a terminal trust confirmation; unattended private-CA
installation requires the approved pin. With no --listen, join accepts only a
single unambiguous local IPv4 address.
Verify¶
On each installed Authority:
curl --fail --silent http://127.0.0.1:9090/readyz
gdsgate authority keys status
gdsgate authority keys sync root status
Expected: readiness, the same current keyring revision and root epoch, and a separate owner on each host. Restart the new instance and repeat these checks; it must start without another permission. Preserve its entire private state.
On a test installation, verify that an incorrect fingerprint refuses join before permission redemption. Stop the first Authority and verify an allowed resource through the public Proxy entry. The surviving Authority takes the write lease; the first host has no permanent master role.
Rotate synchronized keyrings¶
Run on either provisioned Authority as root. Both Authority services must be running, share the current revision and have synchronized clocks. Keyrings hold the material that seals control rows and audit records.
Expected: a successor distribution revision and one prepared control key.
Check keys status on both hosts after at least one poll_secs interval
(default 5 seconds). Both must hold the prepared key before activation:
Expected: a new revision; the prepared key becomes active and the former active key remains for verification. Replicas reload without file copying or service restarts. Activation requires an acknowledgement of the exact prepared revision, no older than 30 seconds, from every mandatory owner.
Negative check in an acceptance window: stop B after prepare and wait more than 30 seconds. Activation on A must fail without changing the canonical revision. Start B, wait for it to load and acknowledge the revision, then repeat activation. Do not remove B from membership to bypass the check.
| Task | Command / constraint | Reference |
|---|---|---|
| Audit HMAC rotation | authority keys rotate audit --prepare, then --activate |
Keyring commands |
| Checkpoint signing rotation | authority keys rotate audit --prepare --kind signing, then --activate; prepare one kind at a time |
Keyring commands |
| Control retirement | authority keys rotate control --retire; existing sealed-row and key-loss guards apply |
Keyring commands |
| Audit retirement | Refused: historical audit keys remain available | Audit verification |
A local edit of the original control.keys or audit.keys does not publish a
new revision while synchronization is enabled.
Rotate the distribution root¶
Run as the local administrator on a provisioned Authority. Each member must have its own receiver.key bound to its stable owner.
Root rotation replaces the credential that encrypts canonical integrity snapshots. It preserves existing control and audit material. To replace those keys, use keyring rotation after the root transition completes.
On the initiating Authority:
Expected: a preparation ID, the initiator, and an acknowledgement state for
every retained owner. Wait until every owner reports acknowledged, then
activate on the same Authority:
gdsgate authority keys sync root rotation activate
gdsgate authority keys sync root status
curl --fail --silent http://127.0.0.1:9090/readyz
Check authority keys sync root status and /readyz on each retained host after polling and lease
handoff. Expected: the next root epoch, no pending preparation, and ready.
Replicas reload without restarting. During local installation, readiness and
new RPC admission are blocked; leadership resumes after successful verification.
Ordinary ring edits, membership additions and bootstrap issuance are blocked
while a preparation is pending.
A root-status read can report a revision conflict while the running Authority
holds its protected local cache lock. Repeat only
authority keys sync root status after a short pause, then check the reported
epoch, acknowledgements and readiness before another administrative operation.
A refused status read does not establish that an earlier activation failed.
Withdraw an Authority¶
Retain at least two owners. To withdraw from a two-node cluster, first bootstrap a replacement Authority and confirm its readiness. Prepare on an owner that will remain in the cluster:
gdsgate authority keys sync root rotation prepare \
--withdraw authority-c
gdsgate authority keys sync root status
gdsgate authority keys sync root rotation activate
Repeat --withdraw for each explicitly reviewed removal. Only retained owners
receive the new root and contribute required ACKs. A missing retained owner
blocks activation; a timeout never withdraws it. After commit, a removed
Authority exits and refuses restart with its former membership. Disable its
service and remove its addresses from caller failover lists and any Authority LB.
Withdrawal prevents future root delivery to that owner. It does not revoke previously known CA, control or audit keys, or separate DB/KMS credentials. If the host was compromised, rotate or revoke those credentials through their respective procedures.
Cancel or recover a preparation¶
If preparation cannot finish, run on any retained Authority:
Expected: the same root epoch and no pending preparation. Cancellation advances
the material revision, invalidates the old preparation and lets replicas remove
its local journal. If the initiator is lost, cancel from a retained owner and
prepare again there. Cancellation and activation are serialized; only one can
commit. After an interrupted administrative command, inspect authority keys sync root status
before retrying.
Retained Authorities recover an interrupted committed root/cache write on
restart or the next poll. An old paired backup can catch up through committed
generations using its independent receiver credential. Keep the entire
keyring-sync directory together, including root.key or root.enc,
receiver.key, snapshot.enc and any root-install.json. A wrapped root also
requires its original wrapping provider. Do not delete the cache watermark
to accept a rolled-back database.
Diagnose and recover bootstrap¶
Read the affected instance's service log before changing its private state:
| Symptom | Check / action |
|---|---|
| Trust refusal | Verify the entry, approved fingerprint and TLS name; retain TLS verification |
| Expired permission before verified delivery | Issue a new permission for the same cluster, owner and role. Repeat the original join with the same state and local inputs; receiver keys and nonce are retained. Authority verifies existing membership before accepting replacement proof |
| Lost response or interrupted local write | Retry the original join inputs; saved nonce, receiver keys and verified response permit recovery |
| Working-directory or state permission refusal | Inspect path ancestors and rerun service install authority with the saved configuration |
| Invalid distribution state or rollback | Stop the affected instance; restore database and protected state as a paired set |
| Rotation cannot activate | Restore each retained member and wait for its acknowledgement of the prepared revision |
| Leader discovery failure | Check shared database access, current membership and fenced lease |
An installed join --if-missing validates saved bindings without spending a
permission. Do not delete its watermark or receiver key to bypass a refusal.
A new empty database creates a new cluster unless paired state is restored.
Existing client anchors and enrolled identities cannot join it by restarting.
Back up the shared database and every Authority's entire keyring-sync
directory together. Preserve root.key or wrapped root.enc, receiver.key,
snapshot.enc, watermarks and any pending installation journal. A wrapped root
also requires its original provider. See paired recovery.
Add another Authority¶
Repeat join with the new host's individual address. Any admitted Authority can issue the permission. The public entry routes bootstrap through a surviving Proxy even after the first host is lost. New membership and keyring revisions reach existing Authorities and Proxies automatically.
An Authority LB, when used, receives the new ready backend through deployment tooling. The public Proxy LB also needs a new backend when adding a Proxy. These balancers are external to gdsgate.
Choose the bootstrap endpoint¶
Use the same public Proxy address used by clients and Connector join. The verified
bundle identifies registration and protected Authority bootstrap routes. No
proxy. prefix is added to the supplied domain, and no registration port or
Authority peer is inferred from an unverified response.
Individual Authority peers remain required for control forwarding. An optional private Authority LB changes caller routing; it does not change bootstrap, recipient identities, keyring synchronization or rotation acknowledgements.
Migrate an existing cluster¶
Do not run the new-deployment setup prepare command over an existing cluster.
Take a protected recovery backup and schedule the restart of its Authorities.
On the existing first Authority, retain the current store, OIDC, policy,
listener and certificate settings. If keys still use legacy per-domain files,
run authority keys legacy migrate before enabling sync. Enable [ha] and [ha.keyring_sync] on the initializer, assign its stable owner,
and declare its individual HTTPS control address in [ha.authority_peers]. Keep both existing integrity ring
references as the initializer's seed.
Start the first Authority and verify its canonical revision. Add new instances with join --role authority and the verification steps above.
Keep an existing instance's owner and private state when upgrading it. Local edits of old ring files are no
longer a distribution mechanism. Keep the paired backup until every instance
has passed restart and keyring checks. Rollback requires restoring consistent
database and integrity material; switching off sync while reusing stale seed
files is not a safe rollback.
Profile and limits¶
| Setting | Behavior |
|---|---|
| Security profile | The first setup selects it; joined roles receive the shared minimum |
| Production key generation | ha.keyring_sync.generation_kms_key names a locally declared provider |
| Automatic distribution | Integrity rings, protected root transitions and authenticated Authority membership |
| Operator-owned inputs | Local provider credentials, external TLS files, firewall rules and LB backends |
| Withdrawal | Root rotation can explicitly withdraw owners; removing a peer address does not revoke trust |
| Recovery | Database and protected per-instance state form a paired backup |
Field syntax and limits: keyring synchronization reference.