Change and release runbooks

These are the canonical change procedures. Every planned change records its approval, backup point, expected exit, and rollback boundary before execution. Use the flat operations.md headings only as compatibility forwarders.

Release verification

Procedure gate

Gate	Contract
Classification	Read-only
Prerequisites	The tagged release artifacts, approved public verification key, target platform, and the change record.
Owner	Release operator or supply-chain verifier.
Approval	None to verify; deployment approval is required after all checks pass.
Backup point	Preserve the prior running artifact and deployment manifest before replacement.
Command or check	Run the signature, checksum, version, and SBOM checks below against the exact artifact set.
Expected status and exit	Every invoked verifier exits 0, and the inspected version, target, signature, checksum, and SBOM agree with the release record. No deployment starts on a failed or mismatched check.
Stop condition	Stop on a missing signature, unexpected signer, checksum mismatch, wrong tag, wrong target, or SBOM mismatch.
Recovery	Quarantine the artifact set, retain the verification output, and obtain a fresh signed set from the release owner.
Rollback	Keep the current deployment unchanged. Roll back only to the last independently verified artifact.
Escalation	Escalate signature or provenance failures to security and release engineering.

Re-verify what will run. For highest assurance, rebuild from the tagged commit and compare the binary checksum with the published checksum.

Preflight verification signatures

Procedure gate

Gate	Contract
Classification	Read-only
Prerequisites	cosign, the release public key, checksum file, checksum signature, and artifact directory.
Owner	Supply-chain verifier.
Approval	None for verification.
Backup point	Preserve the downloaded checksum and signature before replacing them.
Command or check	cosign verify-blob --key <release-public-key> --signature <checksums-signature> <checksums-file> followed by sha256sum -c <checksums-file>.
Expected status and exit	Both commands exit 0. The second command reports every listed artifact as OK.
Stop condition	Stop on any non-zero exit, missing artifact, or signer mismatch.
Recovery	Delete or quarantine the set and redownload from the approved release channel.
Rollback	N/A; no running service was changed.
Escalation	Release security owner.

Bind the verification key to release tooling rather than a developer's local key. A checksum match without an authenticated checksum file is not release verification.

Reproducing the build

Procedure gate

Gate	Contract
Classification	Planned change
Prerequisites	The exact tagged source revision, locked dependencies, pinned toolchain, target, and the release script.
Owner	Release engineering.
Approval	Change approval before using the rebuilt artifact for deployment.
Backup point	Preserve the published artifact and its verification record.
Command or check	Check out the exact tag in an isolated checkout and run ./scripts/release.sh; compare its generated checksum with the signed published checksum.
Expected status and exit	The script exits 0; the rebuilt binary and published checksum match byte-for-byte for the same target.
Stop condition	Stop on source, dependency, toolchain, target, path-remap, or checksum drift.
Recovery	Recreate the isolated checkout from the tag and record the toolchain and environment difference.
Rollback	Deploy the prior independently verified artifact, not an unmatched local rebuild.
Escalation	Release engineering and supply-chain security.

The release recipe uses the pinned toolchain selected by CI, sets SOURCE_DATE_EPOCH from the tagged commit, and applies path-remap flags before producing the binary, checksums, and SBOM. The recipe does not promise a fixed timezone, locale, or unrelated container image.

SBOM

Procedure gate

Gate	Contract
Classification	Read-only
Prerequisites	The signed CycloneDX SBOM, release version, and verified target-specific binary.
Owner	Supply-chain verifier.
Approval	None for inspection; approval is required before accepting a mismatch as an exception.
Backup point	Preserve the SBOM with the artifact verification record.
Command or check	jq -e '.metadata.component.name, .metadata.component.version, (.components | length)' <release-sbom>.cdx.json; compare the component name and version with the release record.
Expected status and exit	jq -e exits 0, the component is gdsgate at the release version, and the component list is non-empty. A malformed or mismatched SBOM is a release failure.
Stop condition	Stop deployment on a malformed, missing, or mismatched SBOM.
Recovery	Obtain the SBOM paired with the signed artifact set.
Rollback	Keep the current artifact until a matching SBOM is available.
Escalation	Supply-chain owner.

The SBOM filename is versioned but not target-specific. Verify the target from the signed binary filename and release record; do not infer a target from the SBOM metadata.

Surviving a redeploy

Procedure gate

Gate	Contract
Classification	Planned change
Prerequisites	A verified replacement artifact, persistent store, node identity directories, CA trust, audit keys, and a maintenance window.
Owner	Service owner and deployment operator.
Approval	Change approval and backup confirmation before stopping the service.
Backup point	Back up the store, external audit keys, node state_dir, and deployment manifest immediately before replacement.
Command or check	Run gdsgate --config <config> doctor, then use the replacement's explicit service command, such as gdsgate --config <connector-config> connector.
Expected status and exit	Doctor exits 0; the replacement stays active, preserves the identity, and later passes the verification procedure. A startup failure exits non-zero.
Stop condition	Stop if the store is in-memory or unreachable, identity material is missing, schema is incompatible, or readiness does not become 200 after the role sets it.
Recovery	Keep the old artifact and state, restore the previous process, and investigate without spending a new enrollment token.
Rollback	Restart the last verified artifact against the unchanged persistent store and state directories. Do not downgrade a store schema that is ahead of the binary.
Escalation	Service owner for identity or readiness, database owner for store, security for key loss.

Three things must outlive the process:

State	Source location	If it is ephemeral
Authority store	store_url, defaulting to the persistent file under the home state directory	The CA, registry, audit chain, catalog, and request state are lost or replaced.
Node identity	[enroll].state_dir	The key and leaf are lost. A spent join token is not a recovery path.
Renewal path	[enroll].renew_endpoint, derived from the Authority endpoint when unset	A valid identity cannot renew if its control-plane route is unavailable.

The background renewal loop renews over the existing mTLS identity only. It never falls back to a spent enrollment or join token. A lost state directory requires a deliberate fresh enrollment with a newly minted token.

One-time token exhaustion

Procedure gate

Gate	Contract
Classification	Incident action
Prerequisites	Authority access, confirmation that the old token is spent or exposed, a persistent replacement state_dir, and the intended node role.
Owner	Enrollment and security operator.
Approval	Incident or enrollment approval before minting a replacement token; revoke or quarantine the old identity if compromise is possible.
Backup point	Preserve the old node state and registry evidence before deleting or replacing anything.
Command or check	gdsgate --config <authority-config> authority create-token --role <role>; start the node once with the token, then confirm its persistent identity and authority list-nodes --health.
Expected status and exit	Token creation exits 0; the newly enrolled long-running service stays active with persistent identity. Subsequent restarts use mTLS renewal and do not consume another token.
Stop condition	Stop if the node has no persistent state, the role does not match, or the old identity may be compromised. Never retry a spent token against another endpoint.
Recovery	Install the new token through the approved secret channel, persist the state, and verify the registry row.
Rollback	Stop the new node and revoke its identity if enrollment was incorrect. Do not restore or reuse the spent token.
Escalation	Security for token exposure or identity compromise; Authority owner for registry refusal.

systemd

Procedure gate

Gate	Contract
Classification	Planned change
Prerequisites	A systemd unit, protected configuration, persistent StateDirectory, and a bootstrap token delivered outside the unit file where possible.
Owner	Host and service operator.
Approval	Change approval before editing the unit or restarting the service.
Backup point	Save the prior unit, config, identity directory, and journal cursor.
Command or check	systemctl daemon-reload && systemctl restart gdsgate-connector && systemctl is-active --quiet gdsgate-connector; then run the verification procedure.
Expected status and exit	daemon-reload, restart, and is-active exit 0; the state directory remains the same across restart.
Stop condition	Stop if the unit cannot access its persistent state or config, or if readiness and tunnel checks fail.
Recovery	Restore the previous unit and config, reload systemd, and restart the last verified artifact.
Rollback	systemctl stop gdsgate-connector, restore the saved unit/config, systemctl daemon-reload, then systemctl start gdsgate-connector.
Escalation	Host owner for permissions and systemd, service owner for identity or protocol failures.

StateDirectory=gdsgate creates a persistent directory under the system state root. Point [enroll].state_dir at the persistent node directory. A bootstrap token is single-use and should not be left in a long-lived unit file.

Containers

Procedure gate

Gate	Contract
Classification	Planned change
Prerequisites	A verified image, mounted configuration, persistent state volume, external store or mounted store volume, and a deployment rollback revision.
Owner	Container platform and service operator.
Approval	Change approval before replacing the image or volume mapping.
Backup point	Snapshot the store and persistent identity volume; save the prior manifest.
Command or check	docker compose up -d connector or the equivalent platform rollout command, followed by docker compose ps connector and the verification procedure.
Expected status and exit	The platform reports the container running; the process keeps its identity and the role checks pass. A restart must not create a new registry identity.
Stop condition	Stop if state is mounted on the writable container layer, the store is in-memory, or the connector re-enrolls unexpectedly.
Recovery	Restore the prior manifest and persistent volume mapping. Quarantine any newly created identity for review.
Rollback	Roll back the image and manifest while retaining the persistent state and store backup.
Escalation	Platform owner for volume or rollout failure, Authority/security for a new or revoked identity.

Mount configuration and identity material read-only where possible. Persist the store and audit keys separately. Expose only listeners required by the role.

Verifying

Procedure gate

Gate	Contract
Classification	Read-only
Prerequisites	The replacement service is running, its config path, metrics address if enabled, and a controlled test resource.
Owner	Service operator.
Approval	None for health and controlled-session checks; approval is required before using a production resource for the test.
Backup point	Preserve the pre-change doctor report, version, metrics snapshot, and audit cursor.
Command or check	gdsgate --config <config> doctor; curl -sS -o /dev/null -w '%{http_code}\n' "http://${METRICS_ADDR}/healthz"; the same for /readyz; then gdsgate --config <operator-config> authority list-nodes --health.
Expected status and exit	Doctor exits 0, liveness is 200, readiness is 200 only after the role sets it, and the node report shows the expected state. A controlled session opens and closes with an audit event.
Stop condition	Stop traffic if doctor fails, readiness remains 503, node health is stale, or the controlled session does not produce the expected audit evidence.
Recovery	Keep the replacement isolated, compare logs and configuration with the saved pre-change report, and restore the prior service if needed.
Rollback	Use the surviving-redeploy rollback with the old artifact and unchanged persistent state.
Escalation	Service, Authority, or database owner according to the failed check.

doctor is a read-only checklist. It can report an in-memory store, an ephemeral state directory, missing CA generation, schema drift, or an expired identity. It exits 2 for a failed check. The HTTP liveness and readiness endpoints are separate from the node health board.

Proxy restart behavior

Procedure gate

Gate	Contract
Classification	Incident action
Prerequisites	Proxy logs and metrics, Connector fleet inventory, and a controlled session owner.
Owner	Proxy operator during the restart incident.
Approval	Incident authority before a deliberate restart; none for observing an unplanned restart.
Backup point	Preserve the Proxy logs and metric snapshot before restart or failover.
Command or check	Inspect gdsgate_active_connector_tunnels, reconnect, refusal, and duration metrics; run the controlled-session check after the Proxy is ready.
Expected status and exit	Existing Proxy sessions end. Connectors redial with backoff and full jitter; the active tunnel gauge returns to the expected fleet and a new controlled session succeeds.
Stop condition	Stop admission if tunnels remain absent, refusals rise, or the controlled session cannot open.
Recovery	Wait for the reconnect loop within its bounded backoff, then inspect refusal reasons and Authority reachability.
Rollback	Do not attempt session handover. Roll back the Proxy artifact or configuration only after capturing the incident evidence.
Escalation	Proxy/network owner for relay failures; Connector owner for registration failures; Authority owner for authorization failures.

Proxy session state is in memory. No session handover occurs between Proxy processes. Connector control-plane tunnels reconnect automatically; a tunnel that stays up for at least 20 seconds resets its backoff schedule. The gauges and logs are the implemented signals; readiness alone does not prove a tunnel.

Renewing a listener certificate

Procedure gate

Gate	Contract
Classification	Planned change
Prerequisites	The listener's configured certificate and key file pair, a validated replacement pair, filesystem permissions, and a rollback copy of the old pair.
Owner	TLS or service operator.
Approval	Change approval from the TLS owner before replacing a public or enrollment pair.
Backup point	Keep the prior certificate and key pair together until the new pair is verified.
Command or check	Atomically install the new pair, then systemctl reload gdsgate-proxy or kill -HUP <gdsgate-pid>; perform a new TLS client handshake against the listener.
Expected status and exit	Reload exits 0 when the service manager accepts it. The next handshake presents the new pair; established sessions are not renegotiated.
Stop condition	Stop if the pair cannot be parsed, the key does not match the certificate, or the listener logs listener certificate not reloaded.
Recovery	Leave the previous pair in force, restore file ownership and permissions, and validate the pair offline.
Rollback	Replace the certificate and key together with the saved prior pair, send SIGHUP, and repeat the handshake check.
Escalation	TLS owner for material errors, service owner for listener failure, security for key exposure.

The file-backed listener pairs are:

Listener	Pair
Proxy client-facing	[proxy].public_cert_file and public_key_file
Proxy browser origin	[proxy].web_cert_file and web_key_file
Proxy clientless browser	[proxy].webapps_cert_file and webapps_key_file
Authority registration	[authority].enroll_cert_file and enroll_key_file

The listener rereads a certificate and key as a pair. A malformed or mismatched pair leaves the previous working pair in force. SIGHUP rereads listener pairs only. It does not reload the TOML configuration, policy, listen addresses, transport identity, or trust bundle.

The clientless browser entry behind a front

Procedure gate

Gate	Contract
Classification	Planned change
Prerequisites	A front that sends PROXY protocol, an approved [proxy].webapps_proxy_protocol_from list, the browser certificate pair, and a controlled browser test.
Owner	Proxy and front operator.
Approval	Change approval from both front and service owners before enabling the list.
Backup point	Save the prior proxy config and front routing configuration.
Command or check	Validate the front configuration, restart or reload the front as approved, then open the browser entry from a listed and an unlisted source and inspect the Proxy log.
Expected status and exit	A listed source with a valid header is served. A listed source without a header, or an unlisted source with or without a header, is closed.
Stop condition	Stop before exposing the route if the front list is wrong, the header is missing, or an unlisted source is accepted.
Recovery	Restore the prior list and front configuration, then retest direct and fronted paths.
Rollback	Disable the setting and front together; do not leave only one half enabled on a live listener.
Escalation	Front/network owner for header routing, Proxy owner for parsing, security for unexpected admission.

Set the front addresses in webapps_proxy_protocol_from. The observed source address binds the browser ticket and becomes context.src_ip. The default nobody is correct for a directly reached entry. An unreadable list stops startup rather than silently shortening the policy.

The public listener behind a front

Procedure gate

Gate	Contract
Classification	Planned change
Prerequisites	A front with PROXY protocol enabled, an approved [proxy].public_proxy_protocol_from list, a valid public listener pair, and a controlled external client.
Owner	Proxy and front operator.
Approval	Change approval from the network and service owners.
Backup point	Save both front and Proxy configurations before changing either.
Command or check	Validate the front's proxy_protocol on setting, apply the paired change, then run a TLS handshake and controlled login from the fronted path.
Expected status and exit	Only a listed source carrying a valid header is served. single_port applies the same source rule to client and Connector tunnel traffic.
Stop condition	Stop if either switch is enabled without the other, if a no-header path is accepted, or if context.src_ip is not the intended front address.
Recovery	Restore the previous pair of configurations and close the public route until both sides agree.
Rollback	Roll back the front and Proxy changes as one change.
Escalation	Network owner for front behavior, Proxy/security owner for source-IP policy effects.

The public listener's source address affects certificate source-IP pinning, context.src_ip, audit src_ip, and anonymous rate-limit buckets. The join relay and connector-facing listeners are separate and are not covered by this front setting.

Store schema

Procedure gate

Gate	Contract
Classification	Planned change
Prerequisites	A verified binary, store backup, exclusive change window, and a configuration with store_auto_migrate chosen deliberately.
Owner	Database and Authority operator.
Approval	Change and database approval before schema migration.
Backup point	Take a paired store backup immediately before migration and record the binary version.
Command or check	Set store_auto_migrate = false, run gdsgate --config <authority-config> authority migrate --check, then run gdsgate --config <authority-config> authority migrate once for the cluster.
Expected status and exit	migrate --check exits 0 when current and 1 when migrations are pending. migrate exits 0 after applying pending migrations or doing nothing when current. Connectivity, lock, or unknown-applied failures are non-zero.
Stop condition	Stop on an unknown-applied migration, a pending migration outside the window, a failed backup, or a migration error. Pending and unknown-applied are different stop states.
Recovery	For pending state, run the matching migration command in the approved window. For unknown-applied state, use the newer binary or restore the pre-upgrade backup.
Rollback	Migrations are forward-only. Roll back the binary only with a store backup whose schema is not ahead of it. Never delete migration rows.
Escalation	Database owner for store or lock errors, release owner for binary/schema mismatch, Authority owner for service startup.

Each build carries its migrations. With automatic migration on, startup applies pending work after checking for unknown versions. With it off, startup refuses pending work and reports the command. The direct migration command does not require a running Authority and uses the same top-level store_url.

Concurrent migration runs

Procedure gate

Gate	Contract
Classification	Incident action
Prerequisites	The active migration owner, PostgreSQL store access, and the pre-migration backup.
Owner	Database operator.
Approval	Incident or change authority before killing an active writer or retrying after a timeout.
Backup point	Preserve the store state and migration logs before intervention.
Command or check	Run gdsgate --config <authority-config> authority migrate --check; inspect the migration process and PostgreSQL lock state if the command waits.
Expected status and exit	PostgreSQL migration lock wait is bounded at 120 seconds. One winner applies the DDL; a follower waits and then sees no pending work. A timeout exits non-zero with the lock error. SQLite uses its own single-writer behavior and does not use the advisory lock.
Stop condition	Stop after 120 seconds or on an unknown-applied status. Do not run concurrent manual DDL.
Recovery	Identify the active writer, let it commit or roll back, then rerun the read-only check.
Rollback	Do not roll back by deleting migration rows. Restore the paired pre-change backup only under database authority.
Escalation	Database owner and change authority.

Store schema ahead of the binary

Procedure gate

Gate	Contract
Classification	Incident action
Prerequisites	The binary version, MigrationStatus output, and a backup made before the newer binary migrated the store.
Owner	Release and database operators.
Approval	Incident authority before restore or version replacement.
Backup point	Preserve the current store before any attempted recovery.
Command or check	Run gdsgate --config <authority-config> authority migrate --check and gdsgate --config <authority-config> doctor; inspect the unknown_applied report.
Expected status and exit	Both checks report non-zero or a failed check identifying versions this binary does not know. The binary refuses to start or migrate that store.
Stop condition	Stop the older binary. Do not force migration, remove rows, or call the state current.
Recovery	Deploy the newer matching binary, or restore the backup from before the newer migration.
Rollback	Restore the pre-upgrade store and the previous verified binary as a pair.
Escalation	Release owner for version selection, database owner for restore.

The source distinguishes pending, meaning migrations carried by this binary are absent, from unknown_applied, meaning the store contains a version this binary cannot account for. Both are stop states, but only pending work can be applied by this binary.