Health and observability runbooks

These are the canonical health and observability procedures. The flat operations.md route keeps compatibility headings and forwards to these sections. Every procedure has an explicit gate. A command that needs a deployment value uses a shell variable or an angle-bracket placeholder; do not commit a credential, private key, or real endpoint.

Logs

Procedure gate

Gate	Contract
Classification	Read-only
Prerequisites	Access to the service stderr or its platform log sink; RUST_LOG may be set for the next process start.
Owner	The service operator or incident responder who owns the log pipeline.
Approval	None for collection or filtering. Approval is required before changing a shared log policy.
Backup point	Preserve the original log stream or a time-bounded export before filtering or redaction.
Command or check	RUST_LOG=gdsgate_proxy=debug,gdsgate=info,warn gdsgate --config <proxy-config> proxy; set GDSGATE_LOG_FORMAT=json for one JSON object per line.
Expected status and exit	The long-running service stays active; logs go to stderr and service data stays on stdout. A malformed filter or config exits non-zero during startup.
Stop condition	Stop collection if the stream contains a token, private key, full policy, or credential value; preserve the original artifact for the security owner without forwarding it.
Recovery	Restore the approved logging filter and rotate any credential exposed in an unsafe sink.
Rollback	Unset GDSGATE_LOG_FORMAT and return to the deployment's recorded RUST_LOG value. No service data rollback is involved.
Escalation	Escalate suspected disclosure to the security owner; escalate missing or reordered records to the logging-platform owner.

gdsgate writes structured logs to stderr. Stdout is reserved for command data, such as a proxy-ssh byte stream. The default is warn,gdsgate=info: gdsgate* at INFO and everything else at WARN. Setting RUST_LOG replaces that default rather than extending it:

PROXY_CONFIG="<proxy-config>"
CONNECTOR_CONFIG="<connector-config>"
AUTHORITY_CONFIG="<authority-config>"
RUST_LOG=gdsgate_proxy=debug,gdsgate=info,warn gdsgate --config "$PROXY_CONFIG" proxy
RUST_LOG=gdsgate_ssh=trace,gdsgate=info,warn gdsgate --config "$CONNECTOR_CONFIG" connector
RUST_LOG=debug gdsgate --config "$AUTHORITY_CONFIG" authority

Set GDSGATE_LOG_FORMAT=json for one JSON object per line, suitable for Loki, journald, a container log driver, or SIEM ingestion. Any other value, including unset, keeps the human-readable format:

AUTHORITY_CONFIG="<authority-config>"
GDSGATE_LOG_FORMAT=json RUST_LOG=gdsgate=info,warn gdsgate --config "$AUTHORITY_CONFIG" authority

When profile is configured, JSON lines carry a top-level profile field and pretty lines carry profile=<value>. Without it, no line carries that field. The value is stamped on each line, including lines written by background tasks. Collect stderr with the platform pipeline and keep the profile and timestamp when comparing services. Redact tokens, private keys, policy documents, and credential contents before sending a log excerpt to another team.

Metrics and health

Procedure gate

Gate	Contract
Classification	Read-only
Prerequisites	[telemetry].metrics_listen must be configured on the role being probed; the caller needs network access to that protected address.
Owner	The service operator owns role readiness; the observability owner owns scrape and alert rules.
Approval	None for GET probes or scrapes. Approval is required before exposing the listener beyond its protected interface.
Backup point	Export the current scrape and alert configuration before changing it. No service-state backup is needed.
Command or check	curl -sS -o /dev/null -w '%{http_code}\n' "http://${METRICS_ADDR}/healthz", curl -sS -o /dev/null -w '%{http_code}\n' "http://${METRICS_ADDR}/readyz", and curl -sS "http://${METRICS_ADDR}/metrics".
Expected status and exit	GET /healthz returns HTTP 200 while the process serves. GET /readyz returns 503 until this role sets its readiness flag, then 200. GET /metrics returns 200; the curl exit is 0 for a successful HTTP exchange. Unknown paths are 404; non-GET methods are 405.
Stop condition	Stop rollout or traffic admission on a readiness 503, a missing metrics response, a sudden refusal increase, or a metric name not in the source inventory. A liveness 200 alone is not readiness.
Recovery	Keep the role out of rotation, inspect logs and the role health report, and retry the read-only checks after the declared dependency or listener issue is corrected.
Rollback	Restore the previous scrape or alert rule set. Do not hide a failing readiness signal by changing the probe path.
Escalation	Escalate role composition failures to the role owner and scrape or cardinality problems to observability. Escalate suspected access-control or identity failures to security.

The telemetry listener is plain HTTP and carries no credentials. Bind it to loopback or a mesh-internal address, never the public internet. It is absent when metrics_listen is empty. The HTTP surface is operational telemetry only; the audit log is a separate durable, hash-chained sink.

The source contract is deliberately narrow:

• /healthz is unconditional HTTP 200 while the process serves.
• /readyz is HTTP 503 until the role calls its readiness setter, then 200. Authority sets it after its listeners are bound. Proxy sets it at its control/data-plane composition milestone, before later blocking serve work. Connector sets it immediately before its forever-reconnecting serve loop. Connector readiness therefore does not prove that a live tunnel exists.
• There is no Authority public gRPC health service. The custom gate.v1.Health service is hosted by Proxy on its public listener and answers before client authentication. For a TLS public listener, verify the server certificate:

CA_FILE="<ca.pem>"
PROXY_ADDR="<proxy-host>:<proxy-public-port>"
grpcurl -cacert "$CA_FILE" -d '{}' "$PROXY_ADDR" gate.v1.Health/Check

Use grpcurl -plaintext instead when the public listener is deliberately plaintext. The HTTP probe is the common process check. A Proxy health response is a separate anonymous public-surface check and does not establish an Authority control-plane session.

The following marker is the exact union of names registered by describe_metrics() and names emitted by runtime metric calls. It is not a claim that every series has a sample at every scrape:

gdsgate_active_auth_relay_connections gdsgate_active_auth_relay_streams gdsgate_active_connector_tunnels gdsgate_active_sessions gdsgate_authority_relay_connections_total gdsgate_authority_relay_streams_total gdsgate_authorize_duration_seconds gdsgate_authorize_total gdsgate_connector_reconnects_total gdsgate_connector_registrations_refused_total gdsgate_connector_tunnel_disconnects_total gdsgate_connector_tunnel_duration_seconds gdsgate_connector_tunnels_total gdsgate_doctor_check_status gdsgate_doctor_failed gdsgate_doctor_last_run_timestamp_seconds gdsgate_doctor_warnings gdsgate_public_anonymous_refused_total gdsgate_request_authorize_total gdsgate_route_misses_total gdsgate_sessions_total gdsgate_transport_identity_expires_in_seconds gdsgate_transport_identity_renewals_total

At this revision, describe_metrics() registers 16 of those names. Seven are runtime-emitted without an explicit description registration: gdsgate_public_anonymous_refused_total, gdsgate_authority_relay_streams_total, gdsgate_active_auth_relay_streams, gdsgate_authority_relay_connections_total, gdsgate_active_auth_relay_connections, gdsgate_transport_identity_expires_in_seconds, and gdsgate_transport_identity_renewals_total. The checker keeps the documented union exact while exposing that implementation difference.

The per-request gate has exactly these observed kind values:

connect db_connect db_query kube web

Metric	Type and labels	Source-backed meaning
gdsgate_authorize_total	Counter, action, decision	Issuance authorization decisions, including allow, deny, and not_served.
gdsgate_authorize_duration_seconds	Histogram, action	Issuance decision latency.
gdsgate_sessions_total	Counter, protocol	Proxy data-plane sessions opened.
gdsgate_active_sessions	Gauge, protocol	Proxy sessions currently open.
gdsgate_connector_reconnects_total	Counter	Connector tunnel connection cycles that ended or failed before backoff and redial.
gdsgate_connector_registrations_refused_total	Counter	Connector-observed tunnels closed quickly before registration acknowledgement. The Connector cannot label the cause; inspect the Proxy log.
gdsgate_active_connector_tunnels	Gauge	Reverse tunnels registered on this Proxy now.
gdsgate_connector_tunnels_total	Counter, outcome	Connector tunnels by registered or refused outcome.
gdsgate_connector_tunnel_disconnects_total	Counter, reason	Tunnel endings: closed, transport_error, keepalive_failed, evicted, or superseded.
gdsgate_connector_tunnel_duration_seconds	Histogram	How long a registered reverse tunnel stayed open.
gdsgate_public_anonymous_refused_total	Counter, rpc	Public unauthenticated calls refused by the per-IP budget.
gdsgate_route_misses_total	Counter, stage, reason	Requests naming a resource no connected Connector serves; reason is departed or unknown.
gdsgate_authority_relay_streams_total	Counter, outcome	Reverse-tunnel control-plane streams relayed to Authority: open, refused, dial_failed, or unconfigured.
gdsgate_active_auth_relay_streams	Gauge	Reverse-tunnel control-plane streams currently open.
gdsgate_authority_relay_connections_total	Counter, lane, outcome	Public-listener Authority relay connections by renewal or admin lane and open, refused, dial_failed, or unconfigured outcome.
gdsgate_active_auth_relay_connections	Gauge, lane	Public-listener Authority relay connections currently open by lane.
gdsgate_request_authorize_total	Counter, kind, decision	Per-request decisions. kind is kube, web, db_connect, connect, or db_query, not a Cedar action name.
gdsgate_transport_identity_expires_in_seconds	Gauge, role	Seconds remaining on a node's own transport leaf, negative after expiry.
gdsgate_transport_identity_renewals_total	Counter, role, outcome	Transport-leaf renewal outcomes ok or failed.
gdsgate_doctor_check_status	Gauge, id, category	Latest periodic doctor severity: 0 ok, 1 info, 2 warn, 3 fail.
gdsgate_doctor_failed	Gauge	Failing checks at the last periodic doctor run.
gdsgate_doctor_warnings	Gauge	Warning checks at the last periodic doctor run.
gdsgate_doctor_last_run_timestamp_seconds	Gauge	Unix time of the last periodic doctor run.

The gdsgate_doctor_* gauges appear when periodic doctor checking is enabled with [doctor].check_interval_secs > 0. A deployment label from profile is added to every series when configured. Do not add resource ids, node ids, fingerprints, or caller-controlled values as metric labels. Those details stay in structured logs.

Alert on a falling gdsgate_transport_identity_expires_in_seconds together with gdsgate_transport_identity_renewals_total{outcome="failed"}. The background identity renewal path never falls back to a spent enrollment or join token. It keeps the old material while retrying and eventually refuses new handshakes once the leaf is expired.

Connectors on a proxy

Procedure gate

Gate	Contract
Classification	Read-only
Prerequisites	Proxy metrics access, Proxy logs, and the expected Connector fleet list.
Owner	Proxy or fleet operator.
Approval	None for observation. Approval is required before changing a Connector or catalog declaration.
Backup point	Preserve the current metric snapshot and relevant log window before changing configuration.
Command or check	curl -sS "http://${METRICS_ADDR}/metrics", inspect gdsgate_active_connector_tunnels, registration-refusal and reconnect counters, then run gdsgate --config <operator-config> authority list-nodes --health.
Expected status and exit	Metrics returns HTTP 200 and the node-health command exits 0; the active tunnel count and last reports match the expected fleet. A missing node report is an observation, not proof of a dead process.
Stop condition	Stop admitting traffic when active tunnels are unexpectedly zero, refusal counters rise, or a resource has no serving Connector. Do not treat /readyz=200 as proof of a tunnel.
Recovery	Read the Proxy tunnel reason, Connector registration log, catalog declaration, and Authority health report in that order. Retry only after the cause is identified.
Rollback	Restore the prior catalog or Connector configuration from the recorded change; metric reads themselves have no rollback.
Escalation	Escalate transport or Proxy relay failures to the network/Proxy owner, identity or revocation failures to Authority security, and missing resources to the catalog owner.

gdsgate_active_connector_tunnels counts reverse tunnels held by a Proxy. It has no expected value in the exposition because the Proxy has no fleet roster. Compare it with deployment inventory and the Connector logs. A rising gdsgate_connector_tunnels_total{outcome="refused"} with a flat active gauge means registration is arriving and being closed. Common reasons include an unknown or revoked node, an unreachable Authority, a malformed registration, resources owned by another peer, or a catalog that permits none of the Connector's declared resources.

gdsgate_connector_tunnel_disconnects_total distinguishes closed, transport_error, keepalive_failed, evicted, and superseded. A short gdsgate_connector_tunnel_duration_seconds distribution indicates flapping, but the Proxy cannot distinguish a powered-off host from a middlebox under transport_error. The per-Connector fingerprint, node id, and declared label remain in logs, not metric labels.

Distributed tracing (OpenTelemetry)

Procedure gate

Gate	Contract
Classification	Planned change
Prerequisites	A protected OTLP/HTTP collector endpoint and an approved trace retention and redaction policy.
Owner	The observability owner, with the service owner for deployment configuration.
Approval	Change approval is required before enabling export or sending traces to a new collector.
Backup point	Save the previous [telemetry].otel_endpoint and collector routing configuration.
Command or check	Set [telemetry].otel_endpoint = "http://otel-collector:4318", restart the role, then inspect the collector for a trace containing the service name and request correlation.
Expected status and exit	The long-running service stays active and the collector receives OTLP/HTTP spans. A malformed endpoint is a non-zero startup failure; later delivery failure is not an access decision.
Stop condition	Stop export if traces contain tokens, credentials, full payloads, audit-chain material, or an unapproved destination.
Recovery	Unset the endpoint or restore the previous endpoint, restart, and confirm the service continues without export.
Rollback	Restore the saved telemetry configuration and collector rule set. Existing traces already shipped require retention-policy handling; they are not deleted by a service rollback.
Escalation	Escalate redaction or destination issues to security and collector delivery issues to observability.

Tracing is optional and outbound. It dials the collector's OTLP/HTTP receiver, commonly port 4318; it does not bind that port. The W3C traceparent header is propagated over gRPC seams between Proxy, Authority, and Connector. Long-running authority, proxy, connector, and explicit all services export; one-shot CLI commands do not. Spans may carry service.name, profile, action, resource, decision, and session_uid where those fields apply.

Exporter construction validates the endpoint during startup, but successful construction is not proof of later delivery. Confirm receipt at the collector and use its transport diagnostics when spans stop arriving.

The collector hop is plain OTLP/HTTP in this source contract, so keep it on a loopback or mesh-internal path. Traces carry operational metadata and never secrets, full request bodies, or audit-chain material. The data-plane byte relay inside an established tunnel is not traced per message.