HA with direct Authority failover¶
A public L4 LB fronts the active Proxy pool. Each Proxy reaches Authority
instances by individual addresses and falls back to another ready instance.
Two Authorities share PostgreSQL and run as leader/follower for writes.
Authorities synchronize integrity material through native bootstrap.
Profile: pilot; Connector route: tunneled.
Hosting does not define this topology. Choose the Linux binary or Docker Compose tab below. Roles can share hosts or run on separate hosts.
The image packages the same gdsgate binary and sets gdsgate as its
entrypoint. docker run and docker compose run take its arguments;
docker compose exec includes the executable name. setup, authority,
join, config check and their flags stay the same. The platform supplies
configuration mounts, secrets, persistent directories, network addresses
and the process supervisor.
This composition uses Authority HA, bootstrap and synchronization and Proxy HA. The other Authority routing topology uses the same components.
Before you start¶
The deployment owner supplies the same store, OIDC settings and security profile to all roles, assigns individual identities and opens the selected network paths before enabling LB backends.
| Input | Example | Contract |
|---|---|---|
| Public Proxy entry | 203.0.113.10, or gate.example.com with a public certificate |
Certificates |
| Authority and Proxy instances | A 192.0.2.10, B 192.0.2.11 in the Linux example; named services in Compose |
Individual Authority and Proxy peer addresses |
| Shared PostgreSQL | 192.0.2.30:5432, database and role gdsgate; Compose includes postgres |
Store availability arranged separately |
| OIDC | https://idp.example.com/realms/gdsgate, public client gdsgate, group admins |
Same issuer and client on all roles |
| Administration | Protected operator transfer, synchronized clocks, agreed test resources | Root shell for host provisioning; private state per instance |
For an existing cluster use migration.
Do not run initial setup prepare over its existing state.
flowchart LR
C[Clients and Connectors] --> PLB[Public L4 LB]
PLB --> P1[Proxy A]
PLB --> P2[Proxy B]
P1 <--> P2
P1 --> A1[Authority A]
P1 --> A2[Authority B]
P2 --> A1
P2 --> A2
A1 <--> A2
A1 --> DB[(Shared PostgreSQL)]
A2 --> DB
Open the paths¶
| Listener | Who reaches it | Exposure |
|---|---|---|
Proxy 50061–50063; 50064 for the join relay |
Public LB, clients and Connectors through it | Public entry only |
Authority 50050/50051 |
Authorized registration/control callers and individual Authority peers | Private |
Proxy peer 50065 |
Other Proxies | Private, individually addressed |
Authority and Proxy /readyz |
Their LB health checkers and private monitoring | Private; bind addresses depend on the platform |
Store 5432 |
Authorities | Private |
Use raw TCP passthrough with no TLS termination. Registration and tunnel
listeners accept no PROXY headers. For original source-IP preservation on
50061, configure both the LB and Proxy as described in
Proxy HA; the Compose bundle enables PROXY v2 there.
Run the same CLI on either platform¶
An Authority status check illustrates the packaging boundary. The configured
file path is an input; the gdsgate operation is identical:
The same applies to setup prepare, authority keys sync create-token,
authority create-token, join, config check and key rotation. A native
service starts gdsgate --config <file> authority or proxy; Compose records
those arguments after the image entrypoint. Choose the platform once for
provisioning and keep its persistent state across restarts.
Configure and start¶
Follow the selected tab in order. Both perform the same lifecycle: prepare configuration, initialize A, bootstrap B, enroll Proxies separately, admit ready LB backends, and register Connectors. Stop on a failed trust, store, keyring or readiness check.
This example places one Authority and one Proxy on each of two Linux hosts.
Install the supplied binary at /usr/local/bin/gdsgate on both hosts.
Commands run in a root shell; operator transfer uses account ops.
- Open the topology's private and public paths. Leave public LB backends disabled until the corresponding Proxy is ready.
- Complete bootstrap steps 1–7, using Authority A and Authority B. Replace the store, OIDC and gateway advertisements with your inputs. Only A initializes the group; B receives its rings through its owner-bound permission.
- Check matching revisions on both instances and restart B after removing its one-time bootstrap input, as required by the bootstrap procedure.
- Run Proxy steps 1–3, using
the
setup prepareandsetup changecommands. Keep their individual Authority destinations: each prefers A or B and has the other as control and registration fallback. No Authority LB is provisioned. - Keep Authority health on loopback
9090; use private Proxy health9091for the public LB. Authority, Proxy and their state directories remain separate even when they share a host. - Enable only ready public LB backends. Register a Connector through
https://203.0.113.10:50064with the recorded CA fingerprint. Setconnector.proxy_connection_count = 2before starting it; follow HA Connector installation with the selected entry. - Declare and serve the agreed allowed and denied resources, then run deployment acceptance. An empty catalog before resources are declared is expected.
For a DNS public entry, apply the Proxy certificate and registration settings and update Authority's gateway advertisements. Authority registration TLS remains a private bootstrap input independent of the public certificate.
Compose runs the same binary with configuration mounts, secret files and one persistent volume per identity. The included store and LB containers are a single-host example; use separate availability arrangements for host-failure coverage.
-
Download the deployment files into a new directory:
install -d -m 0750 /opt/gdsgate-ha cd /opt/gdsgate-ha for file in compose.yaml configure.sh postgres-init.sh authority-lb.cfg proxy-lb.cfg env.example env-ip.example ip-only.yaml demo.yaml; do curl -fsS "https://gate.gds.dev/docs/deploy/files/cluster-ha/$file" -o "$file" doneFile Purpose compose.yaml Authorities, Proxies, balancers, store configure.sh Initial configuration postgres-init.sh Separate database administrator and application role authority-lb.cfg Authority TCP LB proxy-lb.cfg Proxy TCP LB env.example DNS-mode environment env-ip.example, ip-only.yaml IP-mode environment and relay port demo.yaml Test backend and Connector -
Select the address mode and edit
.env:IMAGE=registry.example.com/gdsgate:v0.10.0-rc.9 DOMAIN=gate.example.com TLS_MODE=dns AUTHORITY_ROUTING=lb OIDC_ISSUER=https://idp.example.com/realms/gdsgate OIDC_CLIENT_ID=gdsgate ACCESS_GROUP=admins PUBLIC_PORT=50061 TUNNEL_PORT=50062 WS_PORT=50063 FRONT_SUBNET=172.31.240.0/24 FRONT_RANGE=172.31.240.128/25 PROXY_LB_IP=172.31.240.2 ENABLE_DEMO=1IMAGE=registry.example.com/gdsgate:v0.10.0-rc.9 DOMAIN=203.0.113.10 TLS_MODE=ip AUTHORITY_ROUTING=lb OIDC_ISSUER=https://idp.example.com/realms/gdsgate OIDC_CLIENT_ID=gdsgate ACCESS_GROUP=admins PUBLIC_PORT=50061 TUNNEL_PORT=50062 WS_PORT=50063 JOIN_PORT=50064 FRONT_SUBNET=172.31.240.0/24 FRONT_RANGE=172.31.240.128/25 PROXY_LB_IP=172.31.240.2 ENABLE_DEMO=1 COMPOSE_FILE=compose.yaml:ip-only.yamlENABLE_DEMO=1: catalog resourcesha-demo,ha-denied, explicit denial onha-denied. For an existing backend, setENABLE_DEMO=0before initialization. -
Create directories and secrets:
install -d -m 0700 secrets install -d -m 0750 -o 10001 -g 10001 config config/tls install -d -m 0700 -o 10001 -g 10001 admin openssl rand -hex 24 > secrets/store_password openssl rand -hex 24 > secrets/postgres_password chmod 0600 secrets/postgres_password chmod 0755 postgres-init.sh install -m 0600 /dev/null secrets/proxy-a.token install -m 0600 /dev/null secrets/proxy-b.token install -m 0600 /dev/null secrets/authority-b-bootstrap.token install -m 0600 -o 999 -g 999 secrets/store_password secrets/postgres_store_password chown 10001:10001 secrets/store_password chown 10001:10001 secrets/proxy-a.token secrets/proxy-b.token secrets/authority-b-bootstrap.token chmod 0600 secrets/store_passwordThese initialization steps apply to a fresh database volume. The init script runs once; existing volumes retain their database roles.
The
postgres:17image initializes as database administratorpostgres. The init script creates non-superuser application rolegdsgate. Only Authority receives the application password. PostgreSQL publishes no host port; both Authorities reach its private network. The initialization copy of the application password belongs to PostgreSQL uid999; the runtime copy belongs to gdsgate uid10001. Both are0600.Install the certificate pair:
install -m 0644 -o 10001 -g 10001 fullchain.pem config/tls/fullchain.pem install -m 0600 -o 10001 -g 10001 privkey.pem config/tls/privkey.pemFor a corporate CA, install its root at
config/tls/issuer-ca.pem, install that root in the system trust stores of clients and connectors. Pass it with--enroll-cafor Connector registration as shown below. Authority bootstrap uses the separate cluster registration CA.No certificate files. Initialization writes
config/transport-ca.pemand printstransport_ca.fingerprint. Record that value forjoinandconnect.
Before generating configuration, select this topology in .env:
configure.sh batches the same gdsgate setup prepare and setup change
commands, using service names and mounted paths. It generates complete
files for each Authority and Proxy; review them before startup.
-
Start PostgreSQL and generate the configurations once:
docker compose config --quiet docker compose up -d --wait --wait-timeout 90 postgres docker compose run --rm --no-deps configureGenerated file Use config/authority-a.toml,config/authority-b.tomlShared store, policy and OIDC; listeners on container interfaces; separate lease owners config/proxy-a.toml,config/proxy-b.tomlSelected Authority routing, individual peering, trusted Proxy LB address config/control.keys,config/audit.keysInitial rings used only by A to initialize the synchronized snapshot config/policy.cedarInitial policy config/transport-ca.pem, both modesPrivate Authority registration trust anchor; bootstrap verifies enroll.gdsgate.internalA's check must report no errors. Before its first start it can warn that the initial registration certificate lacks the final gateway address; Authority renews that cluster-issued certificate at startup. B is prepared for bootstrap; install its pin and permission before starting it. Public DNS TLS belongs to the Proxies. Private Authority registration uses a cluster-issued certificate in both modes.
[ha] enabled = true max_cluster_sessions = 1024 [ha.authority_peers] authority-a = "https://authority-a:50051" authority-b = "https://authority-b:50051"Per-instance value A B GDSGATE_HA_OWNERauthority-aauthority-bGDSGATE_PROXY_PEER_ADDRhttps://proxy-a:50065https://proxy-b:50065Proxy identity volume proxy-aproxy-bAuthority protected-state volume authority-aauthority-bha.keyring_sync.initializetruefalse -
Start only the first Authority and initialize its administrative identity:
docker compose up -d --wait --wait-timeout 90 authority-a docker compose run --rm --no-deps --entrypoint 'gdsgate' configure \ --config /etc/gdsgate/authority-a.toml authority bootstrap-admin --output-dir /admin docker compose exec -T authority-a gdsgate --config /etc/gdsgate/authority-a.toml \ authority keys statusExpected: A is ready, distribution revision 1, two mandatory Authority members. Do not start B with another initializer. This is the first-Authority step, executed with Compose.
-
Bootstrap B directly from A, before starting the selected LB backends:
On the Docker host, read the trusted CA fingerprint from the initialization output or this exported certificate. Set it on B and issue its permission:
Set
ha.keyring_sync.bootstrap_fingerprintinconfig/authority-b.tomlto that SHA-256 hex value without colons, replacing the placeholder.umask 077 docker compose exec -T authority-a gdsgate --config /etc/gdsgate/authority-a.toml \ authority keys sync create-token --owner authority-b --ttl-secs 600 > secrets/authority-b-bootstrap.token chown 10001:10001 secrets/authority-b-bootstrap.token chmod 0600 secrets/authority-b-bootstrap.token docker compose up -d --wait --wait-timeout 90 authority-b docker compose exec -T authority-b gdsgate \ config check /etc/gdsgate/authority-b.toml --role authority docker compose exec -T authority-b gdsgate --config /etc/gdsgate/authority-b.toml \ authority keys statusExpected: B is ready and reports the same revision as A. Its own volume now holds
keyring-sync/root.keyandsnapshot.enc. Compare both holders after polling, using bootstrap verification.Remove the three bootstrap fields from B's configuration and empty the spent secret file. Keep the placeholder file for the Compose secret mount:
sed -i '/^bootstrap_endpoint =/d; /^bootstrap_fingerprint =/d; /^bootstrap_token =/d' config/authority-b.toml truncate -s 0 secrets/authority-b-bootstrap.token docker compose restart authority-b docker compose exec -T authority-b gdsgate --config /etc/gdsgate/authority-b.toml \ authority keys statusVerify B readiness again before continuing. Keep individual Authority peers with either caller route. The selected topology decides whether Proxies dial
authority-lbor the individual Authority addresses.
Leave the authority-lb service profile disabled. Each generated Proxy
file names both individual Authorities for control and registration;
Proxy A prefers A and Proxy B prefers B. No private LB service is started.
-
Issue separate Proxy tokens, check configurations and start the Proxies:
umask 077 docker compose exec -T authority-a gdsgate --config /etc/gdsgate/authority-a.toml \ authority create-token --role proxy > secrets/proxy-a.token docker compose exec -T authority-a gdsgate --config /etc/gdsgate/authority-a.toml \ authority create-token --role proxy > secrets/proxy-b.token chown 10001:10001 secrets/proxy-a.token secrets/proxy-b.token docker compose run --rm --no-deps --entrypoint gdsgate proxy-a config check /etc/gdsgate/proxy-a.toml --role proxy docker compose run --rm --no-deps --entrypoint gdsgate proxy-b config check /etc/gdsgate/proxy-b.toml --role proxy docker compose up -d --wait --wait-timeout 90 proxy-a proxy-b proxy-lbPreserve each instance's identity volume across restarts. Issue new tokens only when registering new identities.
-
After both Proxy instances are ready, remove their spent token references and empty the mounted input files. Keep the files for the secret mounts:
for node in a b; do docker compose run --rm --no-deps --entrypoint gdsgate configure \ setup change --proxy-config "/etc/gdsgate/proxy-$node.toml" \ --set proxy:enroll.token= truncate -s 0 "secrets/proxy-$node.token" done docker compose restart proxy-a proxy-b docker compose psVerify both readiness checks before registering a Connector. Each Proxy must restart from its own persisted identity without another token.
Register a Connector with the same gdsgate join command as on a native
host. The Compose wrapper supplies its configuration and identity volume:
For an existing backend: authority set-resource, a Connector token, join
--serve, [connector].proxy_connection_count = 2 and service installation.
Commands: HA Connector installation.
For the test database below, keep ENABLE_DEMO=1.
-
Start the database and issue a token:
# Keep the selected TLS overlay when adding the demo services. case "$(sed -n 's/^TLS_MODE=//p' .env)" in dns) export COMPOSE_FILE=compose.yaml:demo.yaml ;; ip) export COMPOSE_FILE=compose.yaml:ip-only.yaml:demo.yaml ;; *) echo 'Set TLS_MODE=dns or TLS_MODE=ip in .env' >&2; exit 1 ;; esac openssl rand -hex 24 > secrets/demo_password chmod 0600 secrets/demo_password docker compose --profile demo \ up -d --wait --wait-timeout 90 demo-postgres docker compose exec -T authority-a gdsgate --config /etc/gdsgate/authority-a.toml \ authority create-token --role connector > secrets/connector.token chmod 0600 secrets/connector.token -
Register through the Proxy LB:
For a corporate CA, add
--enroll-ca /etc/gdsgate/tls/issuer-ca.pem.docker compose run --rm -T --no-deps --entrypoint gdsgate connector \ join gate.example.com --name ha-connector --recording central \ --enroll-endpoint https://gate.example.com:50061 --route tunneled \ --serve ha-demo=postgres@demo-postgres:5432 \ --serve ha-denied=postgres@demo-postgres:5432 \ --output /var/lib/gdsgate/connector/connector.toml \ --state-dir /var/lib/gdsgate/connector --metrics-listen 127.0.0.1:9090 \ < secrets/connector.tokenReplace
<transport_ca.fingerprint>with the initialization value:docker compose run --rm -T --no-deps --entrypoint gdsgate connector \ join 203.0.113.10 --name ha-connector --recording central \ --enroll-endpoint https://203.0.113.10:50064 \ --enroll-fingerprint '<transport_ca.fingerprint>' \ --enroll-sni enroll.gdsgate.internal --route tunneled \ --serve ha-demo=postgres@demo-postgres:5432 \ --serve ha-denied=postgres@demo-postgres:5432 \ --output /var/lib/gdsgate/connector/connector.toml \ --state-dir /var/lib/gdsgate/connector --metrics-listen 127.0.0.1:9090 \ < secrets/connector.token -
Set two Proxy connections, check and start:
rm secrets/connector.token docker compose run --rm --no-deps --entrypoint gdsgate connector \ setup change --connector-config /var/lib/gdsgate/connector/connector.toml \ --set connector:connector.proxy_connection_count=2 --set connector:security.profile=pilot docker compose run --rm --no-deps --entrypoint gdsgate connector \ config check /var/lib/gdsgate/connector/connector.toml --role connector docker compose --profile demo \ up -d --wait --wait-timeout 90 connector
Connect a client¶
On the workstation, connect to the public Proxy entry independently of the server platform. For IP-only TLS, supply the recorded transport CA fingerprint:
CLIENT_CONFIG=~/.gdsgate/ha/client.toml
CA_FINGERPRINT='<approved SHA-256 fingerprint>'
gdsgate connect 203.0.113.10:50061 --fingerprint "$CA_FINGERPRINT" \
--output "$CLIENT_CONFIG"
gdsgate --config "$CLIENT_CONFIG" login
gdsgate --config "$CLIENT_CONFIG" ls
gdsgate --config "$CLIENT_CONFIG" authority policy pull
For a DNS entry with approved public trust, use
gdsgate connect gate.example.com:50061 --output "$CLIENT_CONFIG".
Choose the address mode once. The separate directory keeps this cluster's
configuration and trust anchor separate from an existing client installation.
Use --config "$CLIENT_CONFIG" on every subsequent client command.
The login must use the configured OIDC provider, not the development issuer.
Complete an allowed native operation
and a denied operation with the resources selected for acceptance. The Compose
sample serves PostgreSQL resources ha-demo and ha-denied:
In another terminal, use its database credentials for
psql -h 127.0.0.1 -p 15432 -U demo -d ha_demo -W -c 'SELECT 1;'.
Verify the topology¶
Run these checks in an acceptance window. Restore each stopped component
before testing the next failure. A successful ls is not proof of recovery
of leader-dependent access decisions: open a new allowed resource connection.
| Check | Expected | Recovery |
|---|---|---|
| Authority key status on A and B | Same revision and expected holders | Restore the affected node's store access and protected state |
| Readiness plus policy pull and a new native resource connection | Ready nodes in the pools and working store/leader path | Inspect role logs, store access, trust and routing |
Connector doctor --explain with its actual config |
2 verified distinct Proxy nodes, 2 desired; admitted catalogue entry |
Check LB tunnel affinity, Connector settings and startup diagnostics |
| Stop the Authority write leader; open another allowed native connection | New decisions recover after lease takeover | Check the remaining node's lease, direct peer path and store access |
| Stop Proxy A; open another connection | Public LB selects B; new access succeeds | Correct backend withdrawal and surviving tunnel routes |
| Public client tries private control, peer and health listeners | Network refusal | Correct exposure before proceeding |
| Wrong trust pin during fresh enrollment | Refusal before identity issuance | Restore the approved pin; never disable verification |
| Denied resource, direct backend isolation and audit verification | Results satisfy acceptance | Follow the owning access or audit recovery procedure |
| Stop the preferred Authority while its Proxy keeps running | Control and fresh registration use the other individual destination | Check both fallback lists and shared TLS verification settings |
Run checks and failure injection on the platform¶
On each Authority:
gdsgate --config /etc/gdsgate/authority.toml authority keys status
gdsgate --config /etc/gdsgate/authority.toml doctor
Stop A with systemctl stop gdsgate-authority or
systemctl stop gdsgate-proxy. Restore with systemctl start and verify
readiness before returning it to a pool. On a two-host deployment, also
test loss of both roles on one host during the acceptance window.
On the Docker host:
docker compose ps
docker compose exec -T authority-a gdsgate --config /etc/gdsgate/authority-a.toml authority keys status
docker compose exec -T authority-b gdsgate --config /etc/gdsgate/authority-b.toml authority keys status
docker compose -f compose.yaml -f demo.yaml exec -T connector gdsgate \
--config /var/lib/gdsgate/connector/connector.toml doctor
Stop A with docker compose stop authority-a or
docker compose stop proxy-a. Restore with docker compose start and
verify health before the next failure. This checks process failure on
one host; it does not establish host-failure availability.
Existing streams owned by a stopped Proxy end. Keep outputs, revisions, versions and the selected topology with the deployment record, without tokens or private keys.
Upgrade and drain¶
Withdraw one node at a time and verify new access before proceeding. Use change planning, and the upgrade guide for version-specific changes. Keyring rotation follows the same CLI procedure on either platform; replicas reload without copying ring files.
Drain the Proxy from the external LB and wait for measured withdrawal. Restart or update one service at a time, retaining its private state:
Run readiness and native access checks after each individual restart. Do not run both commands as an availability test without checking between them. Re-admit the Proxy only after it is ready.
From rc.7 or earlier: rc.9 upgrade procedure.
- Set the supplied image version in
.env. - Recreate one Authority, verify access, then recreate the other.
- Drain and recreate one Proxy, verify access, then repeat for the other.
docker compose up -d --no-deps --force-recreate authority-a
docker compose exec -T proxy-a touch /tmp/gdsgate-draining
sleep 15
docker compose up -d --no-deps --force-recreate proxy-a
| Setting / behavior | Value |
|---|---|
| Drain marker | /tmp/gdsgate-draining; /readyz returns 503 |
| Withdrawal interval in the example | 15 seconds; adjust to measured LB propagation |
| Container termination grace | 45 seconds |
| Established streams | End when their owning Proxy stops |
| State to preserve | Store, config, keyrings, administrative identity, individual node volumes |
| Upgrade from a version without shared admission | Authorities first; withdraw old Proxies before enabling the HA pool |
Backup¶
Preserve the same application data on both platforms: shared store, configuration, Authority roots and revision caches, certificates and each node's enrolled identity. Pair Authority protected-state backups with the store backup. See paired recovery.
Back up /etc/gdsgate and each role's private state directory using the
approved backup tooling. Preserve permissions and ownership. Include
keyring-sync/root.key (or provider-wrapped root.enc) and snapshot.enc
from every Authority, the
administrative identity and each Proxy and Connector identity.
| Data | Command / location |
|---|---|
| Shared store | docker compose exec -T postgres pg_dump -U gdsgate -Fc gdsgate > gdsgate.dump |
| Configuration, initial seed rings, certificates | config/; separate from the DB dump |
| Authority distribution roots and revision caches | Both Authority volumes, keyring-sync/root.key (or provider-wrapped root.enc) and snapshot.enc; pair with the DB backup |
| Passwords and tokens | secrets/; protected storage |
| Proxy identities | Separate proxy-a, proxy-b volumes |
| Authority registration certificates, both modes | Separate authority-a, authority-b volumes |
| Connector identity | connector volume |
| Administrative identity | admin/; reissue after expiry |
| Test data | demo-store volume |
docker compose down preserves volumes. After removing identity volumes,
revoke the old node UUIDs and register with new tokens. Paired backup.
Separate hosts and other balancers¶
The topology depends on individually reachable peers and stable owner names, not on co-location or container service names. Substitute real private addresses consistently when moving roles to separate hosts. The store and LB require their own availability arrangements. Kubernetes or Swarm placement must preserve the same component state and routing contracts; use component platform mapping with the corresponding platform guide.
Profile and limits¶
| Boundary | Requirement / behavior |
|---|---|
pilot |
Verified TLS, enrolled node mTLS, OIDC, shared store and integrity material |
production |
Profile checks and a declared generation provider for synchronized key rotation |
| Admission | Same ha.max_cluster_sessions on all Proxies; 1024 by default, maximum 4096 |
| Sessions | Loss of an owning Proxy ends streams; new connections recover through surviving routes |
| Browser listeners | Configure their public TCP frontends and certificates separately |
| Recovery | Store and Authority protected state form a paired set; deleting a watermark is not rollback |