Operations

Local process observation

Available in v0.11.0-rc.14 on an enrolled Linux Connector that starts the SSH programs or caged MCP commands itself. An onward SSH target without a Connector receives no observer. Upgrade the participating components to a release supporting observation before enabling it; older releases do not provide this feature.

0.11.x

Available in v0.11.0-rc.14 on an enrolled Linux Connector that starts the SSH programs or caged MCP commands itself. An onward SSH target without a Connector receives no observer. Upgrade the participating components to a release supporting observation before enabling it; older releases do not provide this feature.

Terminal recording captures the SSH streams. Process observation records program launches, failed launch attempts, child processes and individual task exits. This includes programs executed by a script after it decodes a command. Shell builtins, interpreter instructions and work handed to an existing daemon are not separate program launches. Filenames and arguments do not prove the contents of an executed script or library.

Outgoing TCP connections and the supported file operations are separate opt-ins. Before enabling them, check the service requirements and choose an argument privacy profile. The configuration reference lists every process_observation property, default and restriction.

Enable on a local SSH backend

toml
[[connector.backends]]
kind = "ssh"
resource = "build-host"
login_mode = "connector"
process_observation = { mode = "optional", arguments = "metadata" }

The default mode is off. Configure each backend that needs observation and restart the Connector. Changes to the policy or shared limits require restart; reload refuses them. Terminal recording and audit_command have independent settings. A backend with an onward addr cannot enable process observation. Agent host programs are outside this feature's coverage.

Mode Behavior
off Captures no new process events. Previously saved events still retry delivery.
optional Work may continue when observation is unavailable or loses coverage. The audit reports that condition.
required The Connector refuses startup if required observation cannot initialize. A later capture failure prevents new launches and ends the affected observed work. Local SSH requires login_mode = "connector", session = { kill_on_close = true } and delegated cgroup v2.

required is not supported with login_mode = "host_user". Use optional for observation under a host account, or use Connector login for required mode. Stopping work after a capture failure does not undo actions already performed. The configured stall timeout and process cleanup affect how soon work ends.

Configure the service supervisor to terminate the Connector's whole process tree on failure, including nested groups. A killed Connector cannot perform its own cleanup. See the service requirements.

Observe caged local commands

On an MCP backend with a local command or decider_command, configure its cage and process-observation policy. Events from the local server and each subprocess decider include the parent session_uid, resource and a distinct execution id. The execution purpose identifies mcp_server or decider, so session search can distinguish runs.

Remote MCP servers on addr or url are not observed. A local decider beside a remote server can be observed. Enabling observation without a local command or without a cage is refused.

required needs a strict cage with allow_degraded = false. A launch is refused when the required isolation or process-termination support is unavailable. Known capture failure ends the caged process tree, including detached work. A required decider also refuses its call on capture failure, even if decider_fail_closed = false. Cage restrictions and audit coverage remain separate: collecting an event does not grant a program more access.

Outgoing TCP connection phases

Set tcp = true in the backend's process_observation policy. The default is false. This records outgoing native TCP connect() calls from the observed local process tree. Cage network restrictions still apply.

Action Meaning
process.tcp_attempt A TCP connection was attempted. It can still be denied or fail.
process.tcp_returned The connect call returned zero or an errno. -115 (EINPROGRESS) means the result is still pending.
process.tcp_established The connection reached the established state.
process.tcp_closed_before_established The socket closed before establishment was observed. Its reported socket error can be absent or zero.

Events identify the initiating process, socket and network namespace. The attempt contains the requested address; establishment adds the actual local and peer endpoints. Numeric addresses do not identify a DNS name, application request or translated destination beyond that socket. Argument privacy profiles do not hide these addresses.

Read all available phases and loss indicators together. A missing establishment event does not prove connection failure, and a rejected attempt can have no later state event. Pending connections can delay the final observation summary.

This category excludes UDP, inbound listen/accept, payload contents, connection shutdown after establishment and asynchronous io_uring calls. Work delegated to an existing unobserved daemon is outside coverage. Calls rejected before the socket can be identified can produce no TCP event.

A connection commonly produces three events. Check storage, delivery backlog and loss metrics before enabling TCP observation on a busy backend.

Regular-file writes

Set file_writes = true to record native regular-file write and pwrite completions. The default is false. It applies to the same local SSH or caged command tree; no observer is installed on an onward host.

process.file_write identifies the initiating process, inode/device, requested byte count and returned result. A positive result is the accepted byte count; it does not prove that the bytes reached persistent storage. Zero or a negative errno does not confirm a content change. File contents are not captured.

The recorded path is a snapshot taken before completion, in the initiating process's filesystem view. Rename, hard links and mount changes can make it outdated or give the same inode another name. Paths are limited to 511 bytes. path_state reports captured, not_observed, read_failed or too_long. A missing path or inode identity remains missing; inspect the quality fields. Argument privacy profiles do not hide file paths.

Reads, opens, creation, truncate, rename/unlink and permission changes are not write events. Vectored I/O such as writev, mmap writes, asynchronous I/O and work delegated to an existing unobserved daemon are also outside this category. Some unsupported write paths report tracking loss, but absence of a loss event does not establish coverage of unsupported I/O. Required mode ends work after known loss is detected; it does not prevent an operation already performed.

Each supported write can add an audit event. Check capacity before enabling this category for write-heavy workloads.

Synchronous file changes

Set file_changes = true to include synchronous rename/unlink and requests to change size, mode, uid or gid. The default is false.

Action Meaning
process.file_attributes A file size, permission or ownership request completed.
process.file_unlink An unlink operation completed.
process.file_rename A rename operation completed, including its requested flags and original source/destination identities.

Events contain the initiating process, requested changes, object snapshots and return value: zero or a negative errno. The final mode or owner can differ because of operating-system rules or later changes. Requested uid/gid values are numeric kernel IDs, not inferred login names. A successful result does not prove persistent storage durability.

Objects use inode/device and parent inode/device with component names limited to 255 bytes. These names are not full paths. Hard links can share an inode while having different parent/name pairs. Quality fields identify unreadable or unstable information during concurrent changes. A recorded name can remain after unlink; it does not prove that the path still exists. A destination's inode_state distinguishes an absent object from an unreadable identity. Argument privacy profiles do not hide component names.

Only the supported filesystem operations are observed. An error rejected earlier can produce no event, such as an exclusive rename to an existing name. A retry can produce several events. A size-change request can come from truncate, ftruncate or opening with truncation; it does not prove creation. This category excludes create, rmdir, hard-link/symlink creation, timestamps, extended attributes, file flags, mmap, asynchronous I/O and work delegated to an existing unobserved daemon.

File creation

Set file_creates = true to add process.file_create (SES-0818). The default is false. It records supported regular-file creation operations separately from write and change events.

Each event contains the initiating process, requested parent/name and mode/flags, then the observed creation result. Request and result snapshots have separate timestamps and quality fields. Component names are bounded to 255 bytes and are not reconstructed full paths. Argument privacy profiles do not hide them.

Method Meaning
open_last_lookup Creation during an open. An existing-file open with O_CREAT produces no creation event.
vfs_create Direct synchronous regular-file creation, including regular-file mknod.
vfs_tmpfile Anonymous temporary-file creation. It has a parent but no requested component name; its internal label is not a usable filesystem path.

checkpoint_result = 0 confirms creation at the observed boundary. A later access check or file-open step can still fail: the file can exist even when the caller receives an error. Creation does not prove that data was written or persisted. A negative result leaves creation effects unconfirmed and has no confirmed created object. Do not interpret it as proof that nothing changed. Errors rejected before the observed operation can produce no event.

Unlinking and recreating a name can produce a different inode. Linking an existing anonymous file into a directory does not create another inode. Compare request/result identities, timestamps and quality during concurrent changes. This category does not provide a complete open audit or cover directory, symlink/hard-link creation, asynchronous I/O or delegated-daemon work. Filesystem behavior can vary; check creation events on your filesystem before relying on them for audit.

Kernel and service requirements

Use an enrolled Linux Connector. The current validated environment is x86_64 with kernel 6.8; Linux ARM64 is not yet validated. Windows and macOS do not provide this Linux observer. The kernel must provide BTF and the required BPF facilities. Startup reports missing privileges or unsupported kernel features. No compiler or separate observer executable is needed on the Connector host.

The Connector service needs CAP_BPF and CAP_PERFMON. Install or regenerate its unit with gdsgate service install connector using the configuration that enables observation, then restart it. Editing the backend file does not update an existing unit. Authority, Proxy and client workstations do not need these observation privileges. Programs launched for SSH/MCP do not inherit them.

Required SSH observation also needs delegated cgroup v2 and whole-tree cleanup by the service supervisor. For a custom unit, allow bpf and perf_event_open and configure cgroup delegation; required caged commands additionally need pidfd_open and pidfd_send_signal. See the SSH session settings and cage requirements.

If any requested event category is unavailable, required refuses to provide that observation profile. Under optional, affected work can continue with coverage reported unavailable. Other supported backends can remain observed.

Observation does not protect against root, another process with the Connector account's access, or another privileged actor on the host. It is not a process isolation boundary.

Arguments and raw bytes

Profile Captured arguments
metadata Executable filename and process identity. Argument values are not read. Default.
redacted Argument values are erased before storage. A bounded argument count and quality flags remain. Raw values exist temporarily during capture.
full Bounded raw arguments, which can contain passwords, tokens and other secrets. Explicit opt-in.

Executable filenames are limited to 511 bytes; successful-launch arguments to 16 KiB. Failed-launch argument capture is best effort. Check truncation and read-failure flags before treating a value as complete. JSON represents binary fields as base64; terminal output escapes control characters and non-UTF8 bytes.

For metadata, text output shows arguments=not-collected(metadata). This indicates the configured privacy profile, not a capture read failure.

Choose the profile before enabling capture. The ordinary audit text mask does not additionally redact these arguments. The viewAudit permission and access to the audit store govern who can read them; there is no separate permission for argument values. Retention, backups and exports retain captured arguments. Even metadata can disclose sensitive filenames. These profiles do not hide file paths, network addresses or independently configured terminal/command recordings.

Delivery, bounds and restart

The Connector saves observation events under its enrollment-state directory in process-observations. It keeps undelivered events until Authority confirms storage. A timeout does not erase them; delivery retries do not add duplicate events to the audit.

When delivery is unavailable

Capture can remain healthy while Authority is unreachable. Saved events accumulate locally, so monitor pending records/bytes and available disk space. The shared limits bound capture capacity and local pending data; they do not limit central audit retention.

With default shared settings, one observed process tree can track up to 1024 tasks, hold up to 128 events awaiting collection and retain up to 256 events awaiting delivery. Smaller configured buffers can lower these ceilings. These per-tree ceilings remain even with larger shared buffers. They limit simultaneous activity, not lifetime event totals. TCP and file events use the same capacity as process events.

Exhausting capacity produces coverage loss. Under required, the affected work is ended; under optional, it may continue with degraded coverage. Reduce event volume or increase the relevant limits before retrying. Enabling every file category can substantially increase audit volume.

Restart and recovery

Saved events retry after restart, including events from earlier host boots and when new capture is off. Old pending data consumes part of the shared storage budget. If it leaves too little space for new capture, start with capture off to drain delivery, then restart with the intended policy. Do not reduce limits below the saved backlog.

A complete interrupted local write recovers automatically. With a valid saved checkpoint, an incomplete unpublished write is retained as local evidence while saved events resume delivery. An interrupted acknowledgement does not advance delivery confirmation: the Connector retries from the last saved confirmation. Neither recovery establishes complete capture before the interruption.

Retained evidence consumes the shared journal byte and record budget and is not removed by delivery. Each stream retains at most 32 interrupted writes. The startup log and retained-evidence metrics report their presence. If retained evidence prevents startup, stop the service, preserve the state and move only the retained-* files to a private offline archive before restarting. Keep that archive with access restrictions appropriate for command arguments. Do not move published event files or the saved checkpoint to free space.

A missing or invalid published checkpoint, corrupt complete record or unexpected state file requires operator investigation. Preserve the original state and the logged error; do not delete files to force observation to start. Required observation cannot start with unusable state; optional observation can report unavailable coverage. Older builds that do not recognize retained evidence refuse to open it; retain a compatible build for recovery.

Startup examines at most 32 saved boot directories. Exceeding that bound or finding invalid saved state can prevent recovery. Acknowledged directories are retained; preserve the state before any administrative cleanup.

A crash can lose events not yet saved. A new process.started event marks the restart; it does not establish complete coverage of the preceding interruption. Successful delivery of saved events does not fill that gap.

Updates and resource reassignment

Upgrade Authority to a compatible build before enabling observation or new event categories on Connectors. Follow the upgrade guide for database and binary changes. Turning capture off preserves pending data and does not reverse a database upgrade. Do not remove audit tables or restore an older database over newly recorded events to make a downgrade succeed.

Before moving a resource to another Connector, wait for its pending observation events to be delivered. Events first admitted after the ownership change can be refused; retain the saved state for investigation.

Monitor capture and delivery

Use the Connector's metrics endpoint to distinguish a capture problem from a delivery backlog. Metrics contain no actor, resource, session id or argument labels. Counters cover the running collector's lifetime; gauges are snapshots. They do not establish complete coverage of a crash or privileged tampering.

Metric What to check
gdsgate_process_capture_healthy Whether capture is progressing and local storage is usable. Individual runs can still have recorded loss.
gdsgate_process_capture_progress_age_seconds Time since capture last made progress. Growth can indicate a stall.
gdsgate_process_registrations, gdsgate_process_degraded_registrations Observed process trees still retained and those with known loss.
gdsgate_process_tracked_tasks Tracked tasks, including threads.
gdsgate_process_pending_connections Connections still awaiting a state result.
gdsgate_process_pending_file_calls Writes awaiting completion.
gdsgate_process_pending_file_changes File changes awaiting completion.
gdsgate_process_pending_file_creates Creation operations awaiting completion.
gdsgate_process_attempted_total Event attempts during this collector lifetime.
gdsgate_process_lost_total{kind} Known loss: ring, tracking, quota or journal.
gdsgate_process_pending_records, gdsgate_process_pending_bytes Saved current-boot data awaiting delivery confirmation.
gdsgate_process_durable_through, gdsgate_process_acknowledged_through Saved and delivery-confirmed event positions. A growing difference indicates pending delivery.
gdsgate_process_backlog_pending_records, gdsgate_process_backlog_pending_bytes Pending data from previous boots.
gdsgate_process_retained_records, gdsgate_process_retained_bytes Interrupted writes retained as local evidence for the current boot, consuming journal capacity.
gdsgate_process_backlog_retained_records, gdsgate_process_backlog_retained_bytes Retained evidence from previous boots, also charged to the shared journal capacity.
gdsgate_process_delivery_total{stream,outcome} Delivery results for current/backlog data: acknowledged, transport error or local error.

Healthy capture with a growing backlog indicates a delivery problem. Any increase in loss counters means the corresponding activity needs investigation, even after delivery catches up.

Inspect one session

Shell
gdsgate --config operator.toml authority audit list \
  --session SESSION_UID --action process. --limit 100 --no-pager

Use --json to inspect event details and quality fields. Without the action filter, session search also includes its terminal/session audit events.

For a bounded summary without argument or path values:

Shell
gdsgate --config operator.toml authority audit list \
  --session SESSION_UID --process-summary --limit 1000 --no-pager

Each row identifies a source, collector generation and SSH or MCP execution. ADMITTED says whether registration was found in the selected window. DRAINED means a final drain fact was found; inspect LOSS separately. UNCONFIRMED means no drain was found and does not mean the work is active. FACTS counts only process/network/file facts in this window. CENTRAL SEQ is the latest stored source position for that row, not the Connector's local delivery acknowledgement. Separate unavailable/degraded reports are counted below the table.

The summary uses the same audit permissions and supports --node, --since and --since-ms. Its row limit must be between 1 and 5000; earlier facts can fall outside that window. It does not accept --all, --cursor or additional activity filters. With --json, it prints one snapshot object containing captures, rows_inspected, window_limited and unavailable_reports, with no executable argument or path values. This snapshot does not certify complete capture, current liveness or audit-chain integrity.

Action Interpretation
process.exec A program loaded successfully. This is not its eventual exit status.
process.exec_failed A program-load attempt failed.
process.fork A child task was created.
process.task_exit One task exited; a multithreaded program can produce several such events. Its raw status is not the SSH command's session-level exit code.
process.registered Observation was registered for the work. It does not prove that a program started.
process.drained Tracked work and pending events finished draining. Check its loss counters before assessing completeness.
process.loss Known observation loss.
process.started A new capture run began after startup/restart; preceding crash coverage remains unknown.

Also inspect process.observation availability events and the quality fields on individual events. A zero known-loss count does not prove that all possible host activity was observed.

Audit time is when Authority receives the event. kernel_ns is a monotonic host timestamp; compare it within the same boot. Combine PID with boot id and process start identity to distinguish reused PIDs. Delivery order can differ from event order, and late events can arrive after SSH closes.

Use a compatible Authority for session filtering; the CLI refuses a page that contains another session. For audit integrity, follow the normal audit verification procedure, separately from investigating one session's activity.

Search this documentation edition.