Upgrade guide¶

This guide is the operational companion to the complete historical release reference. It records the source-derived release channel contract and the boundary at which an upgrade may be rolled back.

x86_64-unknown-linux-musl

Current release and channel status¶

The workspace source identifies version 0.10.0-rc.13 on the rc channel. The published RC is v0.10.0-rc.13, which is the target of :rc. The last final release is v0.9.0, which remains the target of :latest.

The channel inventories below describe the publication contract. RC publication requires the full gate and an independent reproducibility check.

Channel	Tag shape	Published status	Gate before publish	Floating container tag
alpha	vX.Y.Z-alpha.N	tagged, signed	no gate	none
beta	vX.Y.Z-beta.N	tagged, signed	no gate	none
rc	vX.Y.Z-rc.N	tagged, signed	full gate + reproducibility	:rc
release	vX.Y.Z	tagged, signed	full gate + reproducibility	:latest

Alpha and beta are published without the full gate. RC and release run the full gate and the independent reproducibility check before publication. Only the rc channel moves :rc, and only the release channel moves :latest. Alpha and beta are requested by their immutable tag.

alpha beta rc release

0.10.0-rc.13|rc|signed|full-gate|rc

The release script produces the following artifact classes. A local dry-run may omit the detached signature, but a tagged CI publication is signed.

binary sbom checksums signature-optional

Rollout class chooser¶

Use this chooser before selecting a tag channel. It is an operational compatibility classification; when a change fits more than one row, use the stricter row.

Change class	Minimum rollout channel	Required boundary
Routine	alpha or beta	No certificate, wire, store, policy, audit-consumer, artifact, or security contract changes; do not imply that the full gate ran
Cert or wire	rc, then release	Roll the compatible Authority, proxy, connector, and consumer fleet together; complete the full gate and reproducibility check
Store	rc, then release	Take the paired store and key backup, run migration as a planned step, and keep the pre-migration restore point
Policy	rc, then release	Validate and migrate the policy vocabulary, preserve the break-glass invariant, and use the approval path
Audit or recording consumer	rc, then release	Check event names, codes, classes, recording format, and downstream parsers before opening traffic
Artifact or channel	rc, then release	Verify target, SBOM, checksum signature, floating-tag behavior, and reproducibility before publication
Security-sensitive	rc, then release	Require the full gate, independent rebuild, paired approval, and an explicit recovery owner

The chooser does not turn an alpha or beta into a tested candidate. Promote a build to rc when the compatibility evidence is required, then use the release channel only after the candidate evidence and recovery rehearsal are complete.

Compatibility and preflight¶

This is the read-only preflight. The release owner records the exact tag, workspace version, target, change id, and store_url before changing a service.

Verify the checksum signature, then the checksums and SBOM: cosign verify-blob --key cosign.pub --signature SHA256SUMS.sig SHA256SUMS, sha256sum -c SHA256SUMS, and jq '.metadata.component.name, (.components | length)' gdsgate-<tag>.cdx.json. Any non-zero result is a stop condition.
Run gdsgate --config authority.toml doctor --json. Exit 0 is a passing preflight; exit 2 names a configuration or state finding to fix before the window. Doctor is read-only.
Set or confirm store_auto_migrate = false for a planned migration, then run gdsgate --config authority.toml authority migrate --check against the same store_url. Exit 0 means no pending migration. A non-zero pending result names the migration ids and is not an incident by itself; an unknown-applied result means this binary is older than the store and is a stop condition.
Run gdsgate --config authority.toml authority verify-audit and preserve the result with the change record. A failed chain verification stops the rollout.
Keep the old binary, configuration, identity directories, store endpoint, and key set available until post-change verification passes.

Paired backup and planned migration¶

This is a planned change, owned by the release operator and approved by the change owner. Stop writes or close the traffic window according to the deployment runbook before taking the final pair.

Take one transaction-consistent backup or snapshot of the persistent store addressed by store_url. Take the external audit-anchor, approval, delegation, revocation, inventory, and CA key material in the same backup generation. Include identity directories and any transport trust material required by the role.
Encrypt the backup, store a copy off the Authority host, record its generation and checksums, and prove that the restore owner can read it. A store-only backup cannot verify historical audit integrity. If either the store or paired keys is absent, mismatched, or unreadable, stop before migration.
Preserve the previous binary, configuration, store_url, and key set. Do not replace only one certificate or key in a pair.
With services stopped or with the approved migration lock, run:

gdsgate --config authority.toml authority migrate --check
gdsgate --config authority.toml authority migrate

The first command changes nothing and exits non-zero when pending work exists. The second opens the store directly, applies pending migrations, and is idempotent. Expected success is an applied-count message followed by a current schema; a second run reports nothing to apply. 5. Run only one planned migration against a shared PostgreSQL store. The store migration lock waits 120 seconds; a timeout is a stop condition to inspect the active writer, not a reason to run a second migration concurrently.

MigrationStatus distinguishes pending migrations, which this binary can apply, from unknown-applied versions, which a newer binary already wrote. An unknown-applied result must use the newer binary or the paired pre-migration restore. Do not delete migration rows, edit migration metadata, or downgrade schema rows.

Verification and recovery boundary¶

This is the post-change verification gate. Start the exact new binary with automatic migration still disabled, then require all of the following:

gdsgate --config authority.toml doctor --json exits 0.
GET /healthz returns 200 and GET /readyz returns 200. A 503 readiness response is not success and traffic stays closed.
gdsgate --config authority.toml authority verify-audit succeeds.
The policy version, catalog version, connector serving state, and audit event for the change match the recorded expectation.
One controlled session and its expected protocol operation succeed, and the relevant recording or audit consumer sees the current event/code shape.

If the binary refuses the store, readiness does not become 200, audit verification fails, or the controlled session shows an incompatibility, stop traffic and keep the old binary stopped or isolated. Restore the paired pre-migration store and keys, restore the previous configuration and binary, verify the audit chain and readiness again, and escalate to the release owner. There is no supported in-place reverse migration. A certificate or wire change requires the compatible fleet to be rolled together; do not mix an old peer with a release that changed its verification contract.

Upgrade notes¶

The v0.8 line uses the current Authority, proxy, connector, and client command families. Replace historical auth terminology with authority only when following a current command or configuration contract. Preserve the existing persistent store_url and identity directories across a service replacement.

Order within a mixed fleet¶

Upgrade every proxy before any authority. From v0.8.23 an authority refuses a connector registration that does not state the kind of each resource it declares, and a proxy older than v0.8.19 forwards no kinds. An authority upgraded ahead of its proxies therefore refuses every connector reaching it through one of them, and the proxy logs auth refused connector registration with the reason registration carries no resource kinds. Upgrading the proxy restores the registration; nothing on the authority side reopens it. The reverse order is safe: a proxy that forwards kinds to an older authority is answered as before.

Connectors before the authority, on a cluster with the SSH gate on¶

Only where gdsgate authority activate has already been run for epoch 1 or above; a cluster that has switched nothing on is unaffected in either order.

From the release that adds it, an authority admits an SSH resource to a connector only against a one-time receipt the connector got from its own activation-lease renewal. A connector older than that sends none, so an authority upgraded ahead of its connectors withholds every SSH resource from each one that has not been upgraded yet, answering gate_not_offered; the connector logs this node declared a resource the cluster is not giving it. Upgrading the connector restores it at its next registration, within a renewal interval. Nothing is lost and nothing has to be reprovisioned, but SSH through those connectors is refused in the meantime.

That is the fail-closed direction on purpose. The recorded capability alone cannot say which process is registering, and a connector rolled back to a binary from before the gate does not clear it, so admitting on the record would admit exactly what the gate exists to refuse.

An empty audit queue before an HA upgrade from v0.10.0-rc.6 or earlier¶

Upgrade only when gdsgate_audit_outbox_depth is 0 on every Authority, or doctor reports audit queue with 0 rows waiting: a row the old build left in the queue is held as unsigned after the upgrade.

gdsgate --config authority.toml doctor

To v0.10.0-rc.13¶

Against rc.10, this version changes distribution-root state and adds Kubernetes discovery-source configuration without adding store migrations. Complete the rc.10 procedure when upgrading from an earlier version. The operator needs administrative access to each Authority host and a recovery point containing the shared store and every node's protected local state.

Verify the artifact signature and checksums using the preflight. Preserve the shared store, keyrings, transport identity and each node's distribution provider, cache and journal together. Preserve receiver.key once it exists. Wrapped roots also require continued access to their KMS key. Keep these private files under their service owner's protected directory.
Upgrade all Authorities before starting any root preparation. Keep CA and keyring rotation paused during the rollout. Each Authority creates an independent receiver credential and binds it to its existing owner. Keep the configured owner and credential; do not copy another node's receiver.
On each Authority host, check doctor --json, /readyz and /healthz. Run authority keys sync root-status with that node's configuration. Confirm the current root epoch, matching material revision and absence of an unexpected preparation. A failed store, membership or leader-route check must give not-ready; restore that dependency before opening traffic.
Upgrade Proxies and Connectors in the maintenance window. Preserve the Connector-before-Authority order when upgrading an activated SSH cluster from versions older than rc.10. Check configured distinct Proxy connections, an allowed session, a denied session, audit verification and central recording delivery.
For in-cluster Kubernetes, use the intended ServiceAccount projection and verify the next token refresh without a Pod restart. API backends need the configured impersonation permissions; discovery sources need only their read-only discovery permissions. Removing permission must refuse requests or withdraw discovery targets after their lease expires.

Root rotation is a separate operation: follow prepare, verify and activate. Every retained owner must provide a durable signed ACK. At least two owners must remain, so add a third Authority before withdrawing one from a two-node cluster. A retained owner can cancel a preparation after loss of the initiator; missing ACKs do not trigger timeout withdrawal. Wait for each retained node's new epoch and readiness before completing the change.

A binary downgrade alone cannot undo receiver bindings, pending transitions or committed root epochs. Stop the changed fleet and use the paired recovery procedure with the matching store and every node's local state. Do not restore only the DB or replace a receiver credential. Before reverting a Connector that uses discovery_sources, restore its compatible configuration as well; the older binary cannot run that source. Root membership withdrawal does not revoke CA, audit/control keys or separate DB/KMS credentials the removed node already held.

To v0.10.0-rc.10¶

Against rc.9, this version adds an Authority bootstrap RPC without adding store migrations. An upgrade from an earlier version must also complete the rc.9 migration procedure.

Preserve a paired store and key backup, each node's transport identity, and the client configuration. With keyring sync enabled, also back up the distribution root and protected cache with the store.
Upgrade the Authority and Proxy fleet in a maintenance window. Keep CA rotation paused until the replicas load the same keyring revision. Follow bootstrap and synchronization to enable automatic distribution; a binary upgrade alone does not enable it.
Upgrade Connectors and verify their configured Proxy connections. Existing tunnels end when their owning process stops. Check a resource session and its recording before reopening ordinary traffic.
Upgrade the client binary, stop the previous resident, and run sudo gdsgate --config "$CLIENT_CONFIG" up on the Linux client host. Verify the system unit uses the original user's identity and that gdsgate --config "$CLIENT_CONFIG" status --explain reports active endpoints.
For direct, adjust native application ports to the new values in status: the default maps 22 to 10022. Setting the offset to 0 retains low ports and requires bind privilege. Verify the owning UID can connect and another unprivileged UID cannot. This isolation guarantee applies only to Linux.

A downgrade does not undo firewall or service changes. Stop the upgraded resident before restoring the earlier binary. Before reverting automatic keyring distribution, preserve the current rings and use the paired restore procedure in Authority recovery.

To v0.10.0-rc.9¶

This release adds store migrations m0078 and m0079. Take the paired store and key backup. Restoring an earlier binary requires restoring that pair. For an upgrade from rc.6 or earlier, also complete the audit queue and CA rotation preflight in the rc.7 procedure.

Withdraw client traffic and stop the older Proxies. Allow established sessions to finish within the maintenance window; remaining streams end when their owner stops.
Pause CA rotation, stop all older Authorities, and upgrade the Authority fleet in one window. Apply migrations with the first upgraded Authority; start the remaining instances with the same store and keyrings.
Configure each Authority with a unique lease owner and the full private peer address map. Verify every Authority is ready before resuming CA rotation. The TCP LB must admit ready followers as well as the leader.
Upgrade all Proxies, preserve their separate identity volumes, and configure their private peer addresses. Check readiness before opening the HA pool.
Upgrade Connectors. Set connector.proxy_connection_count = 2 for HA peer routing. A Connector configured for two connections can accept peer opens while one tunnel is available. The default 1 retains local-session compatibility behavior. Remove source-IP affinity from the tunnel LB pool.

Every role in the HA pool must run the same version. Use the HA configuration and ports for DNS or IP-only access. Keep peer listeners private; preserve the store, keyrings and each node's individual identity.

Verify Connector doctor reports the configured number of distinct Proxy connections. Complete the HA checks, including fresh access after stopping each Proxy and Authority, a policy denial, and authority verify-audit. Existing streams do not migrate between owners.

To v0.10.0-rc.7¶

This release adds store migrations m0075 through m0077. Take the paired store and key backup, stop Authority writes, and apply the migrations as a planned operation. Restore the pair to return to an earlier binary.

Before the first Authority stops, require an empty audit queue as described above. Let every CA rotation already requested finish. Pause scheduled and manual CA rotation for the upgrade window. Upgrade every Authority instance in the same window, then resume rotation after every instance reports ready. An older instance does not enforce this release's leader and wait guarantees for an in-progress rotation.

Upgrade the proxies in the same window as Authority. This release adds policy history and review RPCs and changes how a proxy selects the Authority leader after a write refusal. Upgrade connectors afterwards; their established sessions continue under the existing session rules.

After the fleet is current, run authority verify-audit, confirm gdsgate_audit_outbox_depth remains 0, and exercise one registration renewal and one administrative policy read through a proxy.

To v0.10.0-rc.6¶

On Linux hosts running the shipped systemd units, each role now has a state directory of its own under /var/lib/gdsgate. After installing the new binary, run gdsgate service install <role> as root for every role on the host, then restart the roles. The first run turns /var/lib/gdsgate from a link into a directory and names the installed units that still use StateDirectory=gdsgate. Reinstall each of them before starting it; a unit left on the old line exits with status 238.

Symptom	Cause	What to do
`service install` refuses: `... is outside /var/lib/gdsgate`	The role writes outside the state root.	Move the path under `/var/lib/gdsgate/<role>` and update the configuration.
`service install` refuses: `... is where every role's directory is`	The role writes directly in `/var/lib/gdsgate`.	Use `/var/lib/gdsgate/<role>` for that path.
A unit exits with status 238	The unit still has `StateDirectory=gdsgate`.	`gdsgate service install <role>` for that role.

A connector installed with install-connector.sh is installed again with --state-dir under /var/lib/gdsgate/connector.

Scripts that run gdsgate setup apply as root pass --approve-plan <digest> with the digest --dry-run printed.

Upgrade Authority and the proxies in the same window: authority forget-user is forwarded by the Proxy, and an older Proxy does not forward it.

To v0.9.0-rc.11¶

Before starting the new Authority, add onward_auth to every [[discovery.auto_scopes]] entry whose kinds holds ssh, for example onward_auth = ["certificate"] for targets reached with the certificate gdsgate mints. Without it Authority refuses to start and names the scope. A scope managed with authority set-scope is re-applied with --onward-auth.

Upgrade the proxies in the same window as Authority. A proxy older than Authority drops the new fields from the administrative responses it relays, and authority list-agents then shows - under seen for every agent; read with --route certificate until the proxies are upgraded.

A request to a web resource with a gdsgate-identity header is refused unless the person named in the header is also allowed. Remove the header from scripts that set it.

Upgrading from v0.6.x¶

Back up the store and paired key material, validate the current policy, run the migration check, and deploy the exact target. Verify audit continuity and connector reconnects before opening traffic. Keep a tested restore path for a forward-only schema migration.

Upgrading from v0.3.x¶

Review endpoint and enrollment changes before replacing a node. Re-enroll only when the existing identity cannot renew. Confirm transport-CA fingerprints and resource catalog declarations after the first start.

Upgrading from v0.2.x¶

Treat the upgrade as an identity and data migration. Preserve the store and keys, validate the policy schema, and use the current command names from the CLI reference. Do not copy old listener defaults into a current file without checking [endpoints].

Upgrading from v0.1.x¶

Use an isolated restore rehearsal first. Verify the current binary version, store state, policy, audit chain, and readiness endpoint before connecting a production client.