Available in v0.11.0-rc.14 on an enrolled Linux Connector that starts the SSH programs or caged MCP commands itself. An onward SSH target without a Connector receives no observer. Upgrade the participating components to a release supporting observation before enabling it; older releases do not provide this feature.
Terminal recording captures the SSH streams. Process observation records program launches, failed launch attempts, child processes and individual task exits. This includes programs executed by a script after it decodes a command. Shell builtins, interpreter instructions and work handed to an existing daemon are not separate program launches. Filenames and arguments do not prove the contents of an executed script or library.
Outgoing TCP connections and the supported file operations are separate opt-ins.
Before enabling them, check the service requirements
and choose an argument privacy profile.
The configuration reference
lists every process_observation property, default and restriction.
Enable on a local SSH backend
[[connector.backends]]
kind = "ssh"
resource = "build-host"
login_mode = "connector"
process_observation = { mode = "optional", arguments = "metadata" }The default mode is off. Configure each backend that needs observation and
restart the Connector. Changes to the policy or shared limits require restart;
reload refuses them. Terminal recording and audit_command have independent
settings. A backend with an onward addr cannot enable process observation.
Agent host programs are outside this feature's coverage.
| Mode | Behavior |
|---|---|
off |
Captures no new process events. Previously saved events still retry delivery. |
optional |
Work may continue when observation is unavailable or loses coverage. The audit reports that condition. |
required |
The Connector refuses startup if required observation cannot initialize. A later capture failure prevents new launches and ends the affected observed work. Local SSH requires login_mode = "connector", session = { kill_on_close = true } and delegated cgroup v2. |
required is not supported with login_mode = "host_user". Use optional for
observation under a host account, or use Connector login for required mode.
Stopping work after a capture failure does not undo actions already performed.
The configured stall timeout and process cleanup affect how soon work ends.
Configure the service supervisor to terminate the Connector's whole process tree on failure, including nested groups. A killed Connector cannot perform its own cleanup. See the service requirements.
Observe caged local commands
On an MCP backend with a local command or decider_command, configure its
cage and process-observation policy.
Events from the local server and each subprocess decider include the parent
session_uid, resource and a distinct execution id. The execution purpose
identifies mcp_server or decider, so session search can distinguish runs.
Remote MCP servers on addr or url are not observed. A local decider beside
a remote server can be observed. Enabling observation without a local command
or without a cage is refused.
required needs a strict cage with allow_degraded = false. A launch is
refused when the required isolation or process-termination support is
unavailable. Known capture failure ends the caged process tree, including
detached work. A required decider also refuses its call on capture failure,
even if decider_fail_closed = false. Cage restrictions and audit coverage
remain separate: collecting an event does not grant a program more access.
Outgoing TCP connection phases
Set tcp = true in the backend's process_observation policy. The default is
false. This records outgoing native TCP connect() calls from the observed
local process tree. Cage network restrictions still apply.
| Action | Meaning |
|---|---|
process.tcp_attempt |
A TCP connection was attempted. It can still be denied or fail. |
process.tcp_returned |
The connect call returned zero or an errno. -115 (EINPROGRESS) means the result is still pending. |
process.tcp_established |
The connection reached the established state. |
process.tcp_closed_before_established |
The socket closed before establishment was observed. Its reported socket error can be absent or zero. |
Events identify the initiating process, socket and network namespace. The attempt contains the requested address; establishment adds the actual local and peer endpoints. Numeric addresses do not identify a DNS name, application request or translated destination beyond that socket. Argument privacy profiles do not hide these addresses.
Read all available phases and loss indicators together. A missing establishment event does not prove connection failure, and a rejected attempt can have no later state event. Pending connections can delay the final observation summary.
This category excludes UDP, inbound listen/accept, payload contents, connection
shutdown after establishment and asynchronous io_uring calls. Work delegated
to an existing unobserved daemon is outside coverage. Calls rejected before
the socket can be identified can produce no TCP event.
A connection commonly produces three events. Check storage, delivery backlog and loss metrics before enabling TCP observation on a busy backend.
Regular-file writes
Set file_writes = true to record native regular-file write and pwrite
completions. The default is false. It applies to the same local SSH or caged
command tree; no observer is installed on an onward host.
process.file_write identifies the initiating process, inode/device, requested
byte count and returned result. A positive result is the accepted byte count;
it does not prove that the bytes reached persistent storage. Zero or a negative
errno does not confirm a content change. File contents are not captured.
The recorded path is a snapshot taken before completion, in the initiating
process's filesystem view. Rename, hard links and mount changes can make it
outdated or give the same inode another name. Paths are limited to 511 bytes.
path_state reports captured, not_observed, read_failed or too_long.
A missing path or inode identity remains missing; inspect the quality fields.
Argument privacy profiles do not hide file paths.
Reads, opens, creation, truncate, rename/unlink and permission changes are not
write events. Vectored I/O such as writev, mmap writes, asynchronous I/O and
work delegated to an existing unobserved daemon are also outside this category.
Some unsupported write paths report tracking loss, but absence of a loss event
does not establish coverage of unsupported I/O. Required mode ends work after
known loss is detected; it does not prevent an operation already performed.
Each supported write can add an audit event. Check capacity before enabling this category for write-heavy workloads.
Synchronous file changes
Set file_changes = true to include synchronous rename/unlink and requests to
change size, mode, uid or gid. The default is false.
| Action | Meaning |
|---|---|
process.file_attributes |
A file size, permission or ownership request completed. |
process.file_unlink |
An unlink operation completed. |
process.file_rename |
A rename operation completed, including its requested flags and original source/destination identities. |
Events contain the initiating process, requested changes, object snapshots and return value: zero or a negative errno. The final mode or owner can differ because of operating-system rules or later changes. Requested uid/gid values are numeric kernel IDs, not inferred login names. A successful result does not prove persistent storage durability.
Objects use inode/device and parent inode/device with component names limited
to 255 bytes. These names are not full paths. Hard links can share an inode
while having different parent/name pairs. Quality fields identify unreadable
or unstable information during concurrent changes. A recorded name can remain
after unlink; it does not prove that the path still exists. A destination's
inode_state distinguishes an absent object from an unreadable identity.
Argument privacy
profiles do not hide component names.
Only the supported filesystem operations are observed. An error rejected
earlier can produce no event, such as an exclusive rename to an existing name.
A retry can produce several events. A size-change request can come from
truncate, ftruncate or opening with truncation; it does not prove creation.
This category excludes create, rmdir, hard-link/symlink creation, timestamps,
extended attributes, file flags, mmap, asynchronous I/O and work delegated
to an existing unobserved daemon.
File creation
Set file_creates = true to add process.file_create (SES-0818). The default
is false. It records supported regular-file creation operations separately
from write and change events.
Each event contains the initiating process, requested parent/name and mode/flags, then the observed creation result. Request and result snapshots have separate timestamps and quality fields. Component names are bounded to 255 bytes and are not reconstructed full paths. Argument privacy profiles do not hide them.
| Method | Meaning |
|---|---|
open_last_lookup |
Creation during an open. An existing-file open with O_CREAT produces no creation event. |
vfs_create |
Direct synchronous regular-file creation, including regular-file mknod. |
vfs_tmpfile |
Anonymous temporary-file creation. It has a parent but no requested component name; its internal label is not a usable filesystem path. |
checkpoint_result = 0 confirms creation at the observed boundary. A later
access check or file-open step can still fail: the file can exist even when
the caller receives an error. Creation does not prove that data was written
or persisted. A negative result leaves creation effects unconfirmed and has
no confirmed created object. Do not interpret it as proof that nothing changed.
Errors rejected before the observed operation can produce no event.
Unlinking and recreating a name can produce a different inode. Linking an existing anonymous file into a directory does not create another inode. Compare request/result identities, timestamps and quality during concurrent changes. This category does not provide a complete open audit or cover directory, symlink/hard-link creation, asynchronous I/O or delegated-daemon work. Filesystem behavior can vary; check creation events on your filesystem before relying on them for audit.
Kernel and service requirements
Use an enrolled Linux Connector. The current validated environment is x86_64 with kernel 6.8; Linux ARM64 is not yet validated. Windows and macOS do not provide this Linux observer. The kernel must provide BTF and the required BPF facilities. Startup reports missing privileges or unsupported kernel features. No compiler or separate observer executable is needed on the Connector host.
The Connector service needs CAP_BPF and CAP_PERFMON. Install or regenerate
its unit with gdsgate service install connector using the configuration that
enables observation, then restart it. Editing the backend file does not update
an existing unit. Authority, Proxy and client workstations do not need these
observation privileges. Programs launched for SSH/MCP do not inherit them.
Required SSH observation also needs delegated cgroup v2 and whole-tree cleanup
by the service supervisor. For a custom unit, allow bpf and perf_event_open
and configure cgroup delegation; required caged commands additionally need
pidfd_open and pidfd_send_signal. See the
SSH session settings
and cage requirements.
If any requested event category is unavailable, required refuses to provide
that observation profile. Under optional, affected work can continue with
coverage reported unavailable. Other supported backends can remain observed.
Observation does not protect against root, another process with the Connector account's access, or another privileged actor on the host. It is not a process isolation boundary.
Arguments and raw bytes
| Profile | Captured arguments |
|---|---|
metadata |
Executable filename and process identity. Argument values are not read. Default. |
redacted |
Argument values are erased before storage. A bounded argument count and quality flags remain. Raw values exist temporarily during capture. |
full |
Bounded raw arguments, which can contain passwords, tokens and other secrets. Explicit opt-in. |
Executable filenames are limited to 511 bytes; successful-launch arguments to 16 KiB. Failed-launch argument capture is best effort. Check truncation and read-failure flags before treating a value as complete. JSON represents binary fields as base64; terminal output escapes control characters and non-UTF8 bytes.
For metadata, text output shows arguments=not-collected(metadata). This
indicates the configured privacy profile, not a capture read failure.
Choose the profile before enabling capture. The ordinary audit text mask does
not additionally redact these arguments. The viewAudit permission and access
to the audit store govern who can read them; there is no separate permission
for argument values. Retention, backups and exports retain captured arguments.
Even metadata can disclose sensitive filenames. These profiles do not hide
file paths, network addresses or independently configured terminal/command
recordings.
Delivery, bounds and restart
The Connector saves observation events under its enrollment-state directory
in process-observations. It keeps undelivered events until Authority confirms
storage. A timeout does not erase them; delivery retries do not add duplicate
events to the audit.
When delivery is unavailable
Capture can remain healthy while Authority is unreachable. Saved events accumulate locally, so monitor pending records/bytes and available disk space. The shared limits bound capture capacity and local pending data; they do not limit central audit retention.
With default shared settings, one observed process tree can track up to 1024 tasks, hold up to 128 events awaiting collection and retain up to 256 events awaiting delivery. Smaller configured buffers can lower these ceilings. These per-tree ceilings remain even with larger shared buffers. They limit simultaneous activity, not lifetime event totals. TCP and file events use the same capacity as process events.
Exhausting capacity produces coverage loss. Under required, the affected
work is ended; under optional, it may continue with degraded coverage. Reduce
event volume or increase the relevant limits before retrying. Enabling every
file category can substantially increase audit volume.
Restart and recovery
Saved events retry after restart, including events from earlier host boots and
when new capture is off. Old pending data consumes part of the shared storage
budget. If it leaves too little space for new capture, start with capture off
to drain delivery, then restart with the intended policy. Do not reduce limits
below the saved backlog.
A complete interrupted local write recovers automatically. With a valid saved checkpoint, an incomplete unpublished write is retained as local evidence while saved events resume delivery. An interrupted acknowledgement does not advance delivery confirmation: the Connector retries from the last saved confirmation. Neither recovery establishes complete capture before the interruption.
Retained evidence consumes the shared journal byte and record budget and is not
removed by delivery. Each stream retains at most 32 interrupted writes. The
startup log and retained-evidence metrics report their presence. If retained
evidence prevents startup, stop the service, preserve the state and move only
the retained-* files to a private offline archive before restarting. Keep that
archive with access restrictions appropriate for command arguments. Do not
move published event files or the saved checkpoint to free space.
A missing or invalid published checkpoint, corrupt complete record or unexpected state file requires operator investigation. Preserve the original state and the logged error; do not delete files to force observation to start. Required observation cannot start with unusable state; optional observation can report unavailable coverage. Older builds that do not recognize retained evidence refuse to open it; retain a compatible build for recovery.
Startup examines at most 32 saved boot directories. Exceeding that bound or finding invalid saved state can prevent recovery. Acknowledged directories are retained; preserve the state before any administrative cleanup.
A crash can lose events not yet saved. A new process.started event marks the
restart; it does not establish complete coverage of the preceding interruption.
Successful delivery of saved events does not fill that gap.
Updates and resource reassignment
Upgrade Authority to a compatible build before enabling observation or new event categories on Connectors. Follow the upgrade guide for database and binary changes. Turning capture off preserves pending data and does not reverse a database upgrade. Do not remove audit tables or restore an older database over newly recorded events to make a downgrade succeed.
Before moving a resource to another Connector, wait for its pending observation events to be delivered. Events first admitted after the ownership change can be refused; retain the saved state for investigation.
Monitor capture and delivery
Use the Connector's metrics endpoint to distinguish a capture problem from a delivery backlog. Metrics contain no actor, resource, session id or argument labels. Counters cover the running collector's lifetime; gauges are snapshots. They do not establish complete coverage of a crash or privileged tampering.
| Metric | What to check |
|---|---|
gdsgate_process_capture_healthy |
Whether capture is progressing and local storage is usable. Individual runs can still have recorded loss. |
gdsgate_process_capture_progress_age_seconds |
Time since capture last made progress. Growth can indicate a stall. |
gdsgate_process_registrations, gdsgate_process_degraded_registrations |
Observed process trees still retained and those with known loss. |
gdsgate_process_tracked_tasks |
Tracked tasks, including threads. |
gdsgate_process_pending_connections |
Connections still awaiting a state result. |
gdsgate_process_pending_file_calls |
Writes awaiting completion. |
gdsgate_process_pending_file_changes |
File changes awaiting completion. |
gdsgate_process_pending_file_creates |
Creation operations awaiting completion. |
gdsgate_process_attempted_total |
Event attempts during this collector lifetime. |
gdsgate_process_lost_total{kind} |
Known loss: ring, tracking, quota or journal. |
gdsgate_process_pending_records, gdsgate_process_pending_bytes |
Saved current-boot data awaiting delivery confirmation. |
gdsgate_process_durable_through, gdsgate_process_acknowledged_through |
Saved and delivery-confirmed event positions. A growing difference indicates pending delivery. |
gdsgate_process_backlog_pending_records, gdsgate_process_backlog_pending_bytes |
Pending data from previous boots. |
gdsgate_process_retained_records, gdsgate_process_retained_bytes |
Interrupted writes retained as local evidence for the current boot, consuming journal capacity. |
gdsgate_process_backlog_retained_records, gdsgate_process_backlog_retained_bytes |
Retained evidence from previous boots, also charged to the shared journal capacity. |
gdsgate_process_delivery_total{stream,outcome} |
Delivery results for current/backlog data: acknowledged, transport error or local error. |
Healthy capture with a growing backlog indicates a delivery problem. Any increase in loss counters means the corresponding activity needs investigation, even after delivery catches up.
Inspect one session
gdsgate --config operator.toml authority audit list \
--session SESSION_UID --action process. --limit 100 --no-pagerUse --json to inspect event details and quality fields. Without the action
filter, session search also includes its terminal/session audit events.
For a bounded summary without argument or path values:
gdsgate --config operator.toml authority audit list \
--session SESSION_UID --process-summary --limit 1000 --no-pagerEach row identifies a source, collector generation and SSH or MCP execution.
ADMITTED says whether registration was found in the selected window.
DRAINED means a final drain fact was found; inspect LOSS separately.
UNCONFIRMED means no drain was found and does not mean the work is active.
FACTS counts only process/network/file facts in this window. CENTRAL SEQ
is the latest stored source position for that row, not the Connector's local
delivery acknowledgement. Separate unavailable/degraded reports are counted
below the table.
The summary uses the same audit permissions and supports --node, --since
and --since-ms. Its row limit must be between 1 and 5000; earlier facts can
fall outside that window. It does not accept --all, --cursor or additional
activity filters. With --json, it prints one snapshot object containing
captures, rows_inspected, window_limited and unavailable_reports, with
no executable argument or path values. This snapshot does not certify complete
capture, current liveness or audit-chain integrity.
| Action | Interpretation |
|---|---|
process.exec |
A program loaded successfully. This is not its eventual exit status. |
process.exec_failed |
A program-load attempt failed. |
process.fork |
A child task was created. |
process.task_exit |
One task exited; a multithreaded program can produce several such events. Its raw status is not the SSH command's session-level exit code. |
process.registered |
Observation was registered for the work. It does not prove that a program started. |
process.drained |
Tracked work and pending events finished draining. Check its loss counters before assessing completeness. |
process.loss |
Known observation loss. |
process.started |
A new capture run began after startup/restart; preceding crash coverage remains unknown. |
Also inspect process.observation availability events and the quality fields
on individual events. A zero known-loss count does not prove that all possible
host activity was observed.
Audit time is when Authority receives the event. kernel_ns is a monotonic
host timestamp; compare it within the same boot. Combine PID with boot id and
process start identity to distinguish reused PIDs. Delivery order can differ
from event order, and late events can arrive after SSH closes.
Use a compatible Authority for session filtering; the CLI refuses a page that contains another session. For audit integrity, follow the normal audit verification procedure, separately from investigating one session's activity.