Nodes

A node is the edge runtime that collects from and controls gear wherever it sits: it pulls its worklist from the server, runs it on the spot, and ships results back. This page covers worklist pull, placement, tasks and commands, sessions, inbound demux, the task queue, reachability, and shipping telemetry; the declarative shape it executes lives in templates and collection.

Partial

Built: omniglass node run enrolls (node created server-side, enrollment token minted at POST /nodes/{name}:enroll, exchanged for its NATS credential at POST /nodes:claim), connects outbound-only to the server’s in-process NATS server, pulls its worklist over a og.v1.worklist.<node> request-reply (its enabled tasks plus a config_generation), and heartbeats on og.v1.heartbeat.<node> (the server stamps last_heartbeat_at). Per-node subject isolation is enforced: each node’s NATS credential is permitted only its own <node> subjects. Also built: running tasks (the probes, internal/node/probe.go), shipping telemetry (the JetStream TelemetryBatch ingested by internal/bus/consumer.go), and the raw self-log lane (ADR-0066: internal/node/logs.go, GET /nodes/{name}/logs, the console Self-logs panel), the command wire (a per-node pull, actuated over the driver’s binding, ADR-0136), driver polls on their declared schedule (#822), and, for the line protocol over tcp, tls (#1035), telnet (#1033) and ssh (#1032), held sessions (#816, below) with their listeners and the inbound demux (#817), and the same line protocol as datagrams over udp (#1037), stateless, each endpoint on its own bound socket (udp), and http polls and commands over http and https (#902), stateless, each its own request with the endpoint’s declared credential, the https pin shared with the endpoint’s probe (http and https); an http endpoint whose driver logs in (#1114) has one auth state on its node (the cookie jar, the login’s captured values, a digest challenge), shared by its polls and commands, held in memory alone and dropped when its fingerprint moves, on a terminal refusal, and on restart (logging in over http); and an http endpoint’s server-sent events stream (#1038), held as a session sharing that state, its events routed to the listeners as a line is, resumed from its last event id after a drop (a server-sent events stream). Still Design: the durable JetStream command queue, the session pool knobs and the held transports beyond the line protocol (#1034 (websocket), #1036 (mqtt), #1039 (modbus/tcp), #1040 (bacnet/ip), #1041 (knxnet/ip), ADR-0153), config-generation-driven cache invalidation, node self-telemetry, the node-down sweep, the JetStream publish-ack delivery contract (telemetry publish is fire-and-forget today, #430), tick scheduling and concurrency knobs, and config apply (the :apply flow). The credential is a shared secret (the enrollment token doubles as the NATS password); the decentralized nkey/JWT model is deferred. See implementation status and decision log (ADR-0036).

The node

The node is the edge process (omniglass node run), one per site, or the server itself when no site-local edge exists (see Placement). Identity is bound to node.name (a compromised node cannot impersonate another, identity-access); it holds no config, and its writes are confined to its placement-derived visible_set (the owners of its assigned tasks): node mode, not all-visibility system mode.

A node carries the identity triad with one exception. Its id is principal_id, the immutable primary key every reference stores (a node is the detail row of a principal, and endpoint.node_name holds that uuid whatever its column name suggests). Its name is the operator-typed identifier and fleet address, the NATS subject token and the enrollment identity, and it is the one name with no :rename custom method: moving it would move a live subject, so the PATCH body carries no name at all. Its label is the operator label (console falls back to the name). Its optional location is descriptive, not a scope: a node stays fleet-wide, location clearing if that location is deleted (ON DELETE SET NULL). Descriptive does not mean unguarded: the create and the update resolve that reference within the caller’s own location:read scope, the seam every other placement bind uses, and a location outside it is refused as absent rather than as forbidden (ADR-0089, amended by #705). The console blade is read-edit-save via PATCH /nodes/{name} (Edit primary, gated node:update, editing label, description, location; name read-only; enrolling secondary).

A node is a taggable owner: governed tags whose applies_to includes node bind to it (fleet-wide, all-scope, node:update); effective tags are the platform layer plus direct bindings, no cascade (a node is not a scope tree); the blade carries a Tags panel, the list a Tags column and per-key facet. Decommissioning (DELETE /nodes/{name}, node:delete) hard-deletes, cascading its endpoints, their derived tasks, its node-owned tags and self-telemetry, and its enrollment credential; collected component telemetry is untouched.

Getting its instructions

The node pulls a worklist (the tasks and commands resolved for the components placed on it) over a NATS request-reply config pull, and heartbeats separately on its own subject (the protocol), so liveness tracks independently of the pull.

Target design: the cascade-resolved worklist, tracked in #603

The server, not the template, decides placement (next) and resolves the cascade (config / $var: values, effective interval, credentials) before handing the node concrete work. The node never sees a template, only materialized, resolved task and command instances.

Config propagation (declared change to running node)

Target design: config propagation to a running node (:apply and the generation-driven cache), tracked in #827

An endpoint’s connection config (endpoint, snmp community, http auth header) is a projection of the component’s declared config through its template. The node re-pulls the worklist every tick but caches endpoint config for its process lifetime, so a changed input must propagate: reconcile on the server (a changed declared input, via /components/{name}:apply or a direct config write, re-renders the affected endpoints from the current declared config and upserts them, preserving placement) and invalidate on the node (the worklist reply carries a per-node config_generation, the max updated_at across the endpoints the node polls; an advance drops the node’s endpoint cache and re-fetches this tick, a steady value serves from cache: a real change lands within one tick, no restart). The generation moves at operator-config pace, not telemetry pace: the sample-write path never touches endpoint.updated_at and a no-op re-apply does not advance it, so nodes are never woken for nothing.

The node-server protocol

The edge is outbound-only: a node sits behind NAT, so the server never dials it. A node is a NATS client over the WAN, one authenticated outbound connection (an nkey/JWT credential bound to node.name); everything server-to-node arrives on subjects the node is permitted to consume. Three flows share the connection:

The full node NATS contract, per ADR-0036; delivery guarantees are #430

• Telemetry up: the node publishes TelemetryBatch batches ({samples, labels} plus the (task, ts) envelope, below) to a JetStream raw ingress subject; JetStream acks each publish (at-least-once), a Nats-Msg-Id deduping replays (the admission consumer preserves it when republishing to the trusted stream).
• Control down: a durable JetStream consumer on the node’s command queue, plus worklist-change signals (the config-generation bump); subjects scoped by placement.
• Control up: heartbeat (feeding the node-down sweep), command-execution results (the action-row status), session_log transitions, and the :report self-telemetry, each on its own subject.

Commands: a durable server queue, a stateless edge

A command is issued server-side (the action layer records it and writes intended state, alarms and actions) and dispatched onto a durable server-side JetStream command queue. The edge holds nothing durable: the node pulls from its durable consumer (resuming from its last ack on reconnect), runs, and reports the result, updating the action row. A restart loses no command; the held consumer delivers with no poll latency.

Delivery: at-least-once, idempotent by nature

The node publishes at-least-once, resuming unacked publishes on reconnect (JetStream ack plus Nats-Msg-Id dedup); replay is safe without a separate idempotency layer because everything the edge ships is idempotent by its own key: samples dedup on (series, ts) (an idempotent upsert; the edge stamps ts, so the server is ts-authoritative, reorders out-of-order arrivals, and needs no strict ordering on the wire), and command results are an idempotent status update on a known action row by id. Events are not shipped from the edge: an event is derived server-side (an event_rule over samples, or from a log_line, events), so nothing needs dedup; not re-raising the same event next poll is the alarm model’s job (one stateful open alarm, fire and clear).

Buffering and retention are cascade settings

Target design: the edge buffer, tracked in #430; retention is #417

When the server is unreachable the node buffers in memory, bounded, not durable at the edge. Both the buffer (size, shed policy) and retention are cascade-resolved (cascade) with an install-wide platform binding, overridable down the tree. A full buffer sheds oldest metrics first and surfaces it as a node.buffer sample (depth, drops): visible, never silent.

Credentials at the edge

Target design: credentials at the edge, tracked in #603

Worklist materialization resolves credentials server-side, so device secrets travel to the node (over TLS), held decrypt-on-use: in memory, or encrypted at rest in a scratch dir with the key from the SecretProvider, never persisted in plaintext, scoped to the node’s placement, re-fetched on the config-generation bump.

Enrollment

A node is created server-side first (its node.name and properties), the UI mints a per-node enrollment token, and the node claims its identity on first connect, exchanging the token for its NATS credential (a per-node JWT signed for its nkey, scoped to the subjects its placement allows, identity and access).

Fleet auto-enrollment over a discovery_rule, tracked in #1025

Later, a shared enrollment token plus a discovery_rule auto-enrolls a fleet: the node’s own properties (stable facts, selected ENV) derive its name, editable server-side after deploy; a rollout mints no per-node token.

Placement (ETL, cascaded)

Target design: cascaded placement, tracked in #1024

Collection follows ETL: extract and transform (including the extractor’s Expr transform) default to the edge; the shaped samples load to the server, where resolve / bind / calc / evaluate default to central. Placement is a cascaded property (cascade): placement: central makes the server itself the node target (cloud APIs, SaaS pollers, inbound webhooks). A listener endpoint lives where placement puts it (the on-site node for LAN devices, the server for cloud sources), so a registered callback URL resolves to the placed listener’s address, never a hardcode.

Running tasks

Built for driver tasks: a poll runs its request on its schedule and a listen task claims pushed lines over its endpoint’s held session, both locating and transforming each emit at the edge and landing a collection-failed for anything that does not parse (collection). The Expr extraction and the cascading labels below remain design.

Target design: edge normalization (locate + Expr) and the listen mode, tracked in #603

Per task the node runs the protocol over the endpoint’s connection, then normalizes at the edge: the locate + Expr extraction (collection) produces samples and stamps labels (cascading union + override), keeping the wire bytes as raw only on a parse or validation failure (for collection.failed) or under dev raw-mode. A task runs in one of the two collection modes, poll (on the resolved interval) or listen; a held-open connection is a stateful endpoint transport, not a third task type. Both modes assemble the same telemetry payload (below).

The built endpoint types, their per-task params, and the fixed samples each emits are the collection type catalog (endpoint types and their config); this page covers how the node executes them: reachability gating, sessions, the task queue, tick scheduling.

Sessions

Built (#816): the node holds one session per endpoint whose worklist carries a standing listen task over a transport with a session dialer (the line protocol over tcp, tls, telnet and ssh), or a driver poll whose driver declares a session block or rides a transport with no stateless line, reconciled against every worklist pull and torn down on task removal, re-address, or node shutdown. It establishes, lends its connection to the endpoint’s line polls, recovers with a jittered, bounded backoff, and reports each transition as the session-state property (establishing, up, recovering, down, rejected) plus session-reason and a self-log carrying the reason; there is no session_log table (ADR-0150). The machine’s detail is on collection. Built with #817: the subscription on establish (a listener’s instantiation commands, re-sent on every recover) and the demux on the shared connection (listeners first, answers by reply shape; routing). Built with #1049: authentication on establish, as a layer that gates the session. The worklist carries each driver task’s session block (read off the live driver spec) and the endpoint’s connection fingerprint and published gate (ADR-0158); the node renders the login against the delivered secrets (a credential it lacks refuses the session by reference, never by value), runs it on every connection before any poll, command, or arm, and publishes its verdict as endpoint-authenticated. A terminal refusal reads rejected and the node dials no more: a poll over it fails fast and waits its interval (or runs at the first tick its session is up again, so a fixed credential lands data within a tick), a command fails with the reason, and neither falls back to a throwaway connection. The node gates the endpoint at the refusal’s ladder rung for its fingerprint, across its probes, polls, commands and session, publishes the gate as endpoint-rejection with the refusal’s reason beside it as endpoint-rejection-reason (so a probe’s gate on an endpoint holding no session still says why, #1110), and adopts it back from the worklist after a restart (publishing no reason for a refusal it never saw), so it makes no attempt against an unchanged fingerprint; a rotated secret, a new address, port or transport, a changed login declaration, or an operator’s :retry moves the fingerprint and lifts it (auth and rejection, the gate, ADR-0155). Built with #1035: a tls endpoint’s connections run the TLS handshake before anything else, under the verification mode its params carry (derived from the driver’s tls-mode input). In pin mode the node keeps one public key pin per endpoint, shared by the held session and the reachability probe and carried across a session’s recycle: a key trusted on first use is published as tls-certificate (and published again while the worklist does not yet deliver it back), a pin the worklist delivers that differs (an operator’s :retry with certificate) replaces it and recycles the session, a pin the worklist delivered and then stops delivering (withdrawn) is forgotten, so the next connection trusts on first use (the ssh host key likewise), and a key that differs from the pin is a certificate refusal that gates the transport rung, beneath which the probe runs only the TCP connect (tls, ADR-0160). Still design below: the pool knobs.

Target design: sessions and the session pool, tracked in #603; the held transports beyond the line protocol and the http stream are #1034 (websocket), #1036 (mqtt), #1039 (modbus/tcp), #1040 (bacnet/ip), #1041 (knxnet/ip)

A stateful endpoint (ssh, mqtt, anything held open) becomes a session at runtime: one connection keyed by (node, endpoint), shared by every task and command under it, handshake and auth paid once. A pool holds it open across ticks (reconnect, backoff, keepalive); a listener runtime wakes on its inbound. The socket is ephemeral on the node, which reports lifecycle transitions as the session-state property and its own logs (ADR-0150 replaced the designed session_log rows and session view).

Open question

The exact session lifecycle state enum and pooling parameters (idle timeout, max lifetime, pool size per endpoint, a shared versus dedicated session for a stream).

Lifecycle: establish (connect, authenticate, subscribe if a stream rides the session), operate (pollers and stream events over the held connection, demuxed; built for the line protocol, #817), recover (graceful retry with backoff, surfaced as session-state and a node log, never hammering: a rejected credential risks lockout; a subscription is session-scoped, so a reconnect re-subscribes), teardown (exit cleanly, set up again).

Where failures land. session-state and the node’s logs own connection health (cannot connect, auth rejected, dropped, timeout); the data event owns parse health: a parse failure emits a collection.failed event carrying the raw (plus the action row for commands), surfacing as an alertable collection-health sample. A command timeout can touch both.

Inbound handling on a shared connection

Built for the line protocol (#817): one reader per held connection routes every inbound line, listeners first (a claimed line is never an answer unless it also matches the exchange’s declared reply shape), then the exchange in flight by its reply shape (or, with none declared, the first unclaimed line), then unclaimed (counted, and a collection-failed per tick when the spec says so). This reverses the design’s order below, which tried an outstanding poll’s matcher first (ADR-0151); the precedence and its reasons are on collection.

Target design: the ordered matcher set, tracked in #603

One connection carrying heterogeneous inbound frames cannot assume an arriving frame answers the last command. Frames route through an ordered matcher set: every task contributes a matcher (a poller’s awaited-response shape, a listener/stream’s match: predicate), each frame tested in order, first match winning. While a poll is outstanding its response matcher is tried first, then the standing matchers in declared order, so an event arriving mid-poll falls through to its stream instead of being mis-eaten as the response; where the protocol itself frames responses vs events (xAPI *r vs *e, a request id), framing drives routing and the regex only extracts within the matched frame. An unmatched frame lands as raw (orphan, logged): a missing matcher surfaces, never fails silently.

The component task queue

Built: a standing listen task shares its endpoint’s held session with the endpoint’s polls and commands, demuxed by one reader (#817); the per-component lanes and the smart-wait gate below remain design.

Target design: the component task queue, tracked in #603; the durable command queue is ADR-0036

The node’s work is the component task queue (distinct from the central rule engine; workers): poll tasks (produce samples) and command tasks (from run actions, producing a caused event + action-row status), split by shape:

• discrete tasks (pollers, commands): request/response, serialized into per-component lanes. Component, not host, is the contention key: a server with two IPs is one component, and a reboot takes out both endpoints. A shared poller runs once on its parent and fans out at binding.
• standing receivers (listen tasks): always-on, event-driven, not lane-serialized, sharing a held session with pollers (demuxed) or owning their connection.

Smart-wait gate. After a disruptive command the lane blocks until reachability reports the host back up, read from the node’s local copy, not the sample store; a fixed timeout backstops.

Tasks within one endpoint run serially (one probe, then its tasks in order); only distinct endpoints run concurrently.

Open question

Whether to add intra-endpoint concurrency, given that connection and order semantics differ per protocol.

The node-side queue is not durable: durability lives server-side (the JetStream command queue, the cascade-configurable telemetry buffer). On reconnect the node re-pulls its worklist, resumes its durable consumer on the command queue, and replays unacked telemetry publishes (idempotent on (series, ts)).

Implicit reachability

Any endpoint with a host address gets reachability for free: the node pings the host and checks the declared port(s), continuously and out of band; a smart default, bypassable per endpoint (for endpoints that drop ICMP or have no port).

Target design: the layered availability gate, tracked in #603

The results come back as reachable / port_open samples usable in rules and dashboards, and feed the smart-wait gate from the node’s local copy: the connection detector and the dashboard signal are one always-on probe.

The layered availability gate is an OSI-layered set of cheap checks run as a concurrent pre-pass (high concurrency, short timeouts) before a connection-endpoint’s poll tasks. All applicable checks run, each shipping a built-in sample, instanced (the ping by host, the rest by endpoint) and owned by the queried component; the endpoint’s endpoint-reachable verdict is their AND.

Layer	Check	Sample	Notes
L3 network	ICMP ping, batched once per host per tick	icmp-reachable / icmp-rtt-avg	informational (see verdict below); shared by every endpoint on the host
L4 transport	TCP connect (tcp-family)	tcp-open/tcp-connect-time	udp has no presence check: an open udp port cannot be detected, so its transport rung is the driver’s request answered (built, ADR-0163), and the host’s port-unreachable report reads refused
L7 app	protocol handshake: SNMP sysUpTime get (built as snmp’s transport rung, ADR-0821) · SSH handshake+auth · telnet login chain · an HTTP response drawn, after the TLS handshake for https (built, ADR-0902)	(verdict)	the SNMP get is the SNMP liveness (ICMP-independent); SSH/telnet log in only where the driver’s ladder declares a session rung, because their liveness credential can differ from the component’s; the http probe never carries a credential, so an http endpoint’s session rung is proven by its credentialed polls and commands

The verdict respects each layer’s definitiveness. A TCP connect, any L7 handshake and a request answered are definitive, so the ping is informational, everywhere (an ICMP-filtered host still reads up). The UDP presence check this design once leaned on the ping to disambiguate (an ambiguous read timeout, pingGates) is retired: where no port can be opened (udp, and snmp) the transport rung is a request answered. A definitively down layer (TCP refused, UDP ICMP-unreachable, an L7 auth/no-answer) fails the verdict regardless; an inconclusive probe (a setup/resolve error, a missing credential) does not gate.

The per-check toggles retire into the ladder. The params.<name>_check toggles and params.liveness this design proposed are superseded by the ladder a driver declares (ADR-0158): ladder.ping.enabled turns the informational ping off, ladder.transport names the gate (a connect, the protocol’s handshake, or a request answered: the tls handshake shipped as the tls transport’s rung, ADR-0160, and the udp request as udp’s, ADR-0163), and ladder.session.enabled whether a login rung exists. Nothing is toggled per check on an endpoint; the snmp get shipped as snmp’s rung (ADR-0821).

The honest limit, built as stated (#821): a v2c wrong community is a silent drop, so a get failure alone cannot separate down from wrong-community; the rung reads not responsive, timed out, retryable, never rejected. Only an agent that answers authorizationError refuses a community detectably, and that is a terminal auth refusal at the session rung. Cross-referencing the ping narrows it: host pings + get silent = “reachable, SNMP not answering this community, or stopped”; with ICMP fully blocked even that is lost. The ssh probe still tries a params credential every tick (#1060); the built telnet probe never logs in (it reads the first negotiation or banner bytes; the login is the held session’s auth step, so a probe cannot retry a bad credential every tick, ADR-0157). A driver’s ladder.transport.request.get overrides the probe OID.

Poller tasks run only if the verdict is up; listener (mode=listen) tasks run ungated, never pinged; inline probes (icmp/tcp with the host on the task) are the check, ungated. A down endpoint’s gate samples ship in one batched call. L5 (socket) and further L7 handshakes extend the stack as rungs a driver’s ladder declares, not as toggles.

Shipping samples

The node ships a native TelemetryBatch: { samples, labels } plus an envelope (task, batch ts), published to the JetStream raw ingress subject (protobuf-encoded, the proto surviving as the NATS message schema).

Target design: the edge retry buffer, raw on failure, and the OTLP adapter, tracked in #430

The publish is buffered with retry/backoff. On a parse or validation failure the node also ships the raw wire bytes so the server can emit a collection.failed event; on success raw is omitted (there is no telemetry table), unless a dev raw-mode is on. An OTLP adapter at the edge accepts OTLP from third-party tools and translates to the native shape.

Samples are already produced at the edge; an admission consumer binds owner (registry lookup, owner attribution against the node’s placement) at consume time and republishes to the trusted stream the rule engine and persistence read, so a forged owner drops before evaluation, not at the durable write. The server never re-derives observed samples; the node’s job ends at the ship.

Tick scheduling, concurrency, and self-observability

Target design: the tick scheduler, node.self, and the node-down sweep, tracked in #430 (per-task driver poll schedules are built, #822); the seeded node rules are the event_rule (ADR-0050)

A tick groups the worklist by endpoint and runs three phases: the L3 ping pre-pass (batched per host), the gate-verdict pre-pass, the poll phase. The gate pre-passes run at a high fixed concurrency (gateConcurrency), the poll phase across the bounded poll pool (default 16, --workers), so the cheap gate is never throttled by a small --workers and a node facing many dead targets is bounded by concurrency, not the serial sum of probe timeouts (a dead SNMP get costs timeout * (retries+1), via --snmp-timeout / --snmp-retries, default 3s x2). Each poll task gets a per-task deadline (--task-deadline, default 30s).

Settled for driver polls (#822)

A driver poll runs on its declared schedule.every, at heartbeat granularity: the node keeps each poll’s next due time and runs it on the first tick at or after it (a task new to the worklist runs at once, a restart polls at its first tick, and a run whose fetch failed retries on a backoff from the heartbeat, doubling, capped at the interval). Reachability probes still run every tick, and a listener’s pushes are never gated (ADR-0152).

The loop is overrun-aware: it reschedules relative to each tick’s finish, not a fixed ticker; a tick exceeding its interval is flagged and the next fires immediately, so falling behind surfaces rather than silently dropping ticks.

Each tick the node publishes a node.self envelope: tick duration, task attempted/ran/skipped/failed counts, endpoint probed/up/down counts, the node.overrun state. Not special-cased: node.self is node-owned samples (the seeded node.* types) riding the same raw-ingress -> admission -> trusted path; the self shape is built into the binary (no operator-authored template), the admission consumer binds it to the reporting node (owner_kind = node, the node_id arm of the exclusive arc), and the rule engine’s batching and amortized rule refresh apply for free. Self-telemetry is best-effort (a failed report is logged, never fatal).

A node that goes dark reports nothing, so a node-liveness sweep runs server-side: a node whose last heartbeat (or registration, if it never checked in) predates OMNIGLASS_NODE_DOWN_AFTER (default 90s) gets a node-owned node-down alarm, auto-resolved on the next heartbeat. The sweep raises it directly (no event_rule: a dead node emits no sample), keyed by (node-down, node owner) for idempotency across sweeps; this is why the node owner arc reaches event and alarm, not just samples.

A degraded-but-alive node alarms through the ordinary event_rule path: a rule on a node.* key opens a node-owned alarm. Two are seeded, node-overrun (while node.overrun is true) and node-tasks-failing (while node.tasks.failed > 0), both resolving implicitly on the next clean tick, because Evaluate is owner-general (component, system, location, or node), which also unlocks system- and location-owned alarms.