Skip to content
Omniglass Omniglass Omniglass

Data collection

Partial

Endpoints: the device’s APIs, declared once

Section titled “Endpoints: the device’s APIs, declared once”

An endpoint is an API we intend to call on a component (web, qrc, ttp, snmp), not a network endpoint; a host or IP is a variable it consumes, not its identity. It is named by the protocol it speaks, unique within its component. Reachability is the first gate of a ladder (reach, auth, responds, collecting), needing only the transport. The top-level endpoints array declares each once; steps reference one by id, the endpoint owns the connection, and a persistent session outlives any single function run, so subscriptions and inbound streams attach to a connection established once.

endpoints:
  - id: snmp
    type: snmp                     # transport registry entry (param schema)
    host: ${input.ip}              # references INPUTS, not $var: directly (see Inputs)
    version: "2c"
    auth: ${input.snmp}            # snmp-community shape; community field is secret (masked, audited)
    liveness: { oid: 1.3.6.1.2.1.1.3.0 }   # reachability gate, per endpoint
  - id: cli
    type: ssh
    host: ${input.ip}
    credentialRef: ${input.ssh}    # basic-auth shape, bound to a $var: at apply
    persistent: true               # stateful session, outlives function runs
  • Type is the transport, which is the transport, not the protocol driver: the registry covers snmp, http, ssh, telnet, tcp, icmp, webhook, mqtt, syslog, and websocket, each with a connection-param schema and default reachability probe. The protocol handler turning a device’s API into a normalized menu of samples and functions is a separate driver layer, so a device’s OIDs live in its driver, never on a template (ADR-0039).
  • A label is the one identity string an operator types on an endpoint. The name is derived from the type, so every SSH endpoint in the fleet reads ssh: the row says which protocol it speaks and nothing about what it is for. The label (“Control processor”) is where that goes, and it is settable at create rather than on a following edit, an endpoint labelled by a second call being unlabelled at the moment it is made (ADR-0119). It is optional; an endpoint with none reads its derived name verbatim. Note the corollary of a derived name: since the name IS the protocol and is unique within the component, a component holds at most one endpoint per protocol, and a second of the same type is a 409.
  • liveness: the per-endpoint reachability gate deciding whether its functions run (nodes).
  • persistent: true keeps a session open across runs; scheduled functions borrow it to send, listen functions wake on its inbound.
  • Codec and framing: raw-TCP AV control planes wrap payloads (line terminators, length prefixes, NUL framing, JSON-RPC or TTP envelopes); the endpoint’s codec frames outbound and parses inbound to the declared envelope, so a step sees structure, not wire bytes.
  • Node placement is not declared here; it is server-assigned from the component’s location.

Two axes: task mode and endpoint transport

Section titled “Two axes: task mode and endpoint transport”

A task is a node’s unit of collection work, described by two independent axes:

  • Task mode (on the task): poll (we ask) or listen (we wait), stated from our perspective on purpose, since “pull/push” inverts with the frame.
  • Transport (on the endpoint): stateless (a throwaway connection per shot) or stateful (a held-open connection, a session whose state is the session-state property per endpoint, its connect/drop/reconnect reasons in the node’s own logs; ADR-0150).

All four cells are real:

| | poll (we ask) | listen (we wait) | |---|---|---| | stateless | SNMP get, an HTTP request (a driver’s poll over http or https), a udp datagram request | webhook, SNMP trap, syslog, a udp push to the endpoint’s own port | | stateful | SSH-exec or xAPI xStatus on a held session | MQTT subscribe, xAPI feedback, an http server-sent events stream |

Waiting for a frame is one mode (listen) regardless of transport; statefulness lives on the endpoint.

Sessions

Section titled “Sessions”

Built (#816): a session holds its connection across ticks. The node holds one per endpoint whose worklist carries a standing listen task over a transport with a session dialer (the line protocol over tcp, tls (#1035), telnet (#1033) and ssh (#1032), and a server-sent events stream over http and https (#1038)), or any driver poll over tls, telnet or ssh, or a driver poll whose driver declares a session block (a login a throwaway connection would skip, #1049), keyed by the endpoint’s id, which the worklist carries beside its name (endpoint names are unique only per component). The machine runs on its own:

  • establish: the first dial; the session reads establishing until it connects.
  • authenticate: when the driver declares an auth step, it runs on every connection between the dial and anything else (auth and rejection, below): no poll, command, or listener arm rides a connection whose auth step has not passed.
  • operate: the session reads up, and the endpoint’s line polls borrow the held connection instead of dialing per tick. One reader goroutine owns the connection: on an ask it discards whatever already arrived (a banner, a stray line), writes the line, and hands the next line to that ask. An ask whose answer does not come in time recycles the connection, since its late reply would answer the next ask. A pushed line a listener claims is never an answer, and a request that declares its reply shape takes only a line that shape matches (routing, below). A poll issued while the first dial is in flight waits for it (on a budget of its own, then the exchange gets its whole timeout); any other poll over a session with no connection faults at once with a collection-failed naming the state, so a down device never stalls the node’s tick, and it never dials a connection of its own.
  • recover: a held connection that drops (the peer closed, a read failed, the TCP keepalive gave up, an unshaped answer timed out) reads recovering at once, without waiting for a poll, and redials with a jittered exponential backoff (250ms doubling, capped at 30s, 20% jitter). Five consecutive failed attempts read down, where an attempt fails if its dial fails, its connection drops unanswered within 10s, or an answer over it timed out; so a device that accepts and drops, or accepts and never answers, reads down rather than up forever. An answered exchange forgives the failures before it, so a healthy device’s rare missed answer never reads down. The session keeps retrying at the capped pace and leaves down only when a returning connection proves itself (an answer, a listener’s push, or outliving 10s unless answers were what failed), so a failing device publishes no flapping; a stable connection’s drop redials at once.
  • teardown: a listen task that leaves the worklist, a change to what the connection depends on (transport, address, timeout, credentials; a listen task’s spec re-hash keeps it), or node shutdown closes the connection. Teardown publishes no state (the node stopped wanting the session, which says nothing about the device), so the last session-state outlives it and is read by its age.

There is no session table: each transition lands as the session-state property on the endpoint’s component (instance: the endpoint’s name), published at the transition under the listen task, and as a node self-log (facility: session) carrying the reason (ADR-0150). Proven against the simulator’s line protocol: kill the device’s listener mid-session and the session reads recovering, then up once it returns, with the transitions landing on the server.

Auth and rejection

Section titled “Auth and rejection”

Built (#1049). Auth is a layer that gates the session, the held-session form of the ladder the stateless probes climb (dialed, responds, authenticated). Each attempt runs in order: dial, then the auth step, then the listeners’ arm, then up. The auth step is declared by the driver (a login dialog in its session block) or by the transport (a transport that authenticates on its own); its verdict lands as the endpoint-authenticated property (yes or no, published when it changes), the same rung the ssh probe writes. A login that never prompts or a connection that drops mid-login decides nothing: it is an ordinary failed attempt.

One login dialog. A driver declares the prompts to wait for, what to send at each (a secret input’s field, ${secret.<input>.<field>}), and the match that means ready. A prompt need not end a line (login: waits on the same line): a partial line is judged once the component pauses on it, never at an arbitrary read boundary, so a banner arriving in pieces is read whole. A prompt for a step already answered (the component asking again) is a refusal. The same primitive answers a raw tcp password prompt, a telnet login, and ssh keyboard-interactive questions. A credential is only ever written to the wire: a refusal names the reference that was missing, never its value, and any text the component sends is scrubbed of what was sent before it reaches an error or a reason, so an echoing component cannot leak it.

Terminal or retryable. A refusal retrying cannot fix is a terminal error, a class and the component’s own text: the driver declares which replies are terminal (session.terminal, each a match with a class: auth, host-key, certificate, protocol), and a transport classifies its own protocol refusals. A terminal reply during the login, to a poll, a command, or a listener’s arm rejects the session; anything unclassified stays retryable and follows the backoff and budget above.

The login budget. A component that hangs up after the final secret and before ready, with no refusal text, cannot be told from a dropped connection one time; three such logins in a row are an auth refusal (the #1056 ruling). A drop before the secret step stays an ordinary failure, and a login that passes resets the budget.

Rejected. A terminal error moves the session straight to rejected, without spending the budget: it closes the connection and dials no more. Polls fail fast with the refusal (a terminal fault waits the poll’s interval, never the failure backoff), commands fail with the reason, and neither falls back to a throwaway connection, which would retry the bad credential. It leaves rejected only when its connection fingerprint changes (below):

  • anything it connects with changes: the attached secret rotated in the secret store (the endpoint row untouched), the address, port or attached secret (a driver-attached endpoint’s inputs edit in place, PATCH /endpoints/{endpoint} with inputs), the transport, the driver’s login or terminal declaration (re-derived to the endpoint, #823), or its pin. The session is recycled and reconnects within a tick;
  • an operator retries it (POST /endpoints/{endpoint}:retry, gated by endpoint:retry and audited): the endpoint’s retry counter, part of the fingerprint, moves;
  • the driver declares a slow re-probe (session.reprobe, at least 5m; the default is never).

An unrelated edit (a label) moves nothing the fingerprint covers, and a rejected session stays rejected.

Each transition also lands session-reason, the reason as a property (auth: auth rejected when rejected), so an event rule can alarm on it, and the node logs it once, at the transition. Proven over a real socket (linesession_login_integration_test.go: a raw tcp line protocol behind a password prompt; the right password logs in before the first poll, a wrong one is dialed exactly once, never arms, and reads rejected with the component’s text) and over fakes for the machine (session_auth_test.go: a wrong credential dials once over an hour of the session’s clock, a retry dials once more, a re-probe fires on its interval, a transient failure still spends the budget).

In the console (#1059). The component’s reachability read carries each endpoint’s latest session-state with the session-reason beside it, while the endpoint is configured to hold a session (teardown publishes nothing, so a session nobody holds is not served), and the rung and class of its endpoint-rejection (never the fingerprint, which is the node’s to compare). Like every native probe key, a session’s state and reason are the node’s own facts, which the push lane refuses (the push lane refuses a native key), so no push can read a rejected session up or plant a reason the node never scrubbed. So the component’s Collection card, its Endpoints panel and the endpoint’s detail show the session as a pill, the ladder with the held rung marked, and, for a down or rejected session, the reason. A state is stored only on a transition, so the pill is aged by the endpoint’s last_seen, as ADR-0150 requires: a session whose endpoint has delivered nothing for the staleness window reads grey, saying what it last read. A rejected session offers Retry (:retry, gated endpoint:retry, greyed with its reason for a caller without it) and the way into the endpoint’s edit, where pointing the credential at the right secret moves the fingerprint. The same refusal shows whatever met it (#1110): an endpoint holding no session whose reachability probe was refused (a tls certificate, a udp terminal reply, an snmp authorizationError, the ssh probe’s params credential) carries the gate with its reason, endpoint-rejection-reason, which the node publishes beside it, so the notice, its reason and Retry read the same with no session pill beside them, the retry waiting on the gate rather than on a session (ADR-1110). The collection guide teaches it, and the fix-a-rejected-session walkthrough drives it against the simulated panel. A refusal of a changed key (#1078) names both keys, the pinned and the offered, which the gate carries as data, and offers Trust this key: a confirm that teaches why a key changes, then :retry with exactly the offered key it showed as host_key, which the server refuses (409) unless the refusal on record still names it (ADR-0159); the trust-a-changed-host-key walkthrough drives it against the simulated panel’s ssh console.

The connection fingerprint and the layered gate

Section titled “The connection fingerprint and the layered gate”

Built (#1056 ruling, ADR-0158). Every connection to an endpoint, held or stateless (the reachability probe, a throwaway poll or command, the held session), is verified bottom up on the existing ladder:

  • reachable: a ping, informational only. An open, answering transport is what counts, so a component that blocks ping never reads down for it, and a cloud API is never pinged. The driver’s ladder lands icmp-reachable beside the endpoint’s probe (one observation per probe), after the verdict is judged. A layer-2 presence rung below it is #1062.
  • responsive, the transport rung and the gate: a TCP connect on the transport’s port, the protocol’s handshake (the ssh key exchange, tls), or, where an open port cannot be detected (udp, snmp), the driver’s named request answered (snmp’s a standard get). A wrong host, address or port fails here, retryably, and never reads as auth.
  • authenticated, the session rung: the login, attempted only when the transport rung passes.

The ladder is declared in the driver spec (ladder.ping, ladder.transport, ladder.session), its defaults from the transport registry’s rung, so most drivers declare nothing; a rung a transport cannot run refuses at write. A terminal refusal gates its rung and every rung above it, for the endpoint, while the connection fingerprint is unchanged: a digest the server computes over every input a connection is made with (the transport, the declared params, the driver’s inputs, a digest of each attached secret’s sealed envelope, never its plaintext, the session block, the ladder, the ssh and tls pins an operator declared, and the :retry counter). A tls endpoint’s verification mode is one of its driver’s inputs, so editing it moves the fingerprint too. The rungs below keep running, so reachability stays observed while auth is rejected, except where the transport rung itself carries the credential: an snmp get carries the community, so beneath an snmp auth gate nothing is sent and the rung reads down, rejected (snmp). The gate is published as endpoint-rejection (<rung> <class> <fingerprint>, or none; a refusal of a changed key appends <pinned-key> <offered-key>, its evidence, ADR-0159) and carried back on the worklist, so a restarted node makes no attempt against an unchanged fingerprint and keeps the evidence it adopts. Beside it, in the same batch, the node publishes why: endpoint-rejection-reason, the refusal’s class and the component’s own text, scrubbed of anything the node sent, so a gate on an endpoint holding no session, which has no session-reason, still says why it holds. A gate adopted after a restart publishes no reason, since the node never saw that refusal, and the read serves the reason only while a gate holds (ADR-1110). The ssh probe’s params credential rides the same gate (#1060): tried once, then the key exchange runs without it until the fingerprint moves, and never past a pinned key that changed. So does the tls probe: under a certificate gate it runs only the TCP connect, so no handshake reaches a key nobody trusted. A gate for a changed key carries the pinned and offered keys after the fingerprint (responsive certificate <fingerprint> <pinned> <offered>), which the reachability read serves as data (rejection.pinned_key and rejection.offered_key). Proven end to end (run_session_identity_test.go): a silent panel spends the login budget, then a label edit and a node restart try nothing, while a retry, a login-declaration change, a rotated secret, and an ssh attachment each attempt again; a wrong port reads down on refused connects and tries no login, and the port edited in place connects and authenticates within a tick, a label edit through the same route recycles nothing, and the address edited in place moves a healthy session to the new component; the ssh probe sends a wrong password once, across a restart too, and authenticates after a params edit.

Native push: data pushed by smart senders is self-describing (it carries its key), its edge parse a near-identity pass-through marked shape=native; a failed parse keeps the raw on a collection.failed event.

The push lane refuses a native key

Section titled “The push lane refuses a native key”

Every key a native probe writes, its samples (below) and the node’s account of its own connection (the verdict and its rungs, the probe metrics and their timings, a held session’s state and reason, the host key or tls public key it trusted, a tls endpoint’s certificate facts, the gate it publishes), and the collection-failed a task lands when it cannot collect (#1086), is the node’s own observation, and the console reads it as one. So POST /telemetry:push refuses each by name (#1080, ADR-0158 point 5): on every lane, not just its own, under any instance and any casing, and the rejection’s reason names the key. The set is not a second list: it is internal/collection/probekeys, the declaration the probes emit from, where each key is ruled node-only or pushable with the reason it may be pushed. None is pushable today, and a key declared there without a ruling fails its test, as does a key the reachability read serves that is not ruled node-only. A pusher with a measurement of its own, an outside pinger’s round trip say, registers a metric type for it rather than borrowing the node’s.

The push lane is one write surface, and the refusal is no longer its alone: every key has one writer class, and the Storage Gateway holds every write to it (ADR-0161, the ruling). The probekeys set is the node-probe class, so a declared value, a command, a calc rule or an event rule is refused on it the same way (the ssh host key and the tls certificate admit one declared row, the pin set through an endpoint’s :retry), and a push is refused a platform key (health) and a calc rule’s output, naming the rule. On the node’s own lane a node-probe key lands only under the endpoint whose task the node ran: an empty instance is that endpoint’s name, and any other (another node’s endpoint on the same component, a name no endpoint holds) is refused, while a driver’s own keys keep the instance they name, and a node-probe sample dated more than five minutes past the database’s clock is stored at the database’s now, so a node’s account of an endpoint cannot stay current past its tenure. A driver spec may emit only a collected key: a reserved one is refused at the spec’s write (#1035 found the pin case: a driver emitting one would have the worklist hand it back as the pin), and again where the node interprets a task derived before that refusal existed, so a driver cannot speak for the node either.

Endpoint types and their config

Section titled “Endpoint types and their config”

The poll types and listeners in the transport registry and the operator config they read. How the node executes these (tick scheduling, reachability gating, the task queue) is nodes.

Poll protocols and their config

Section titled “Poll protocols and their config”

Built today: the icmp and tcp probes, and the layer-7 http and ssh probes (#812): http draws a real response off the API (any status is an answer), ssh runs a real key exchange and, when the params carry a credential, tries it. The ladder facts each observes: an accepting port whose API never answers is reached but not responsive, and a completed SSH exchange that refuses the credential is responded but not authenticated. A task whose config cannot be used lands a collection-failed event naming the task, never a silent skip. A probe’s config rides the endpoint’s params JSON, unmarshalled on the node:

| transport | shape | config (endpoint.params) | samples | |---|---|---|---| | icmp | inline probe | target (host or IP), count, timeout | icmp-reachable, icmp-rtt-avg (fixed) | | tcp | inline probe | target (host:port), timeout | tcp-open, tcp-connect-time (fixed) | | http | inline probe | target (host:port), timeout | the tcp pair, http-response-time, endpoint-responsive; one GET / with no credential, any status an answer | | https | inline probe | target (host:port), timeout, tls-mode (as tls) | the tcp pair, tls-handshake-time, endpoint-responsive (the handshake is the rung), http-response-time when a response follows, and the certificate facts as tls lands them; no credential | | telnet | inline probe | target (host:port), timeout | the tcp pair, telnet-response-time (the first negotiation or banner bytes), endpoint-responsive; the login is the held session’s auth step, never the probe’s | | ssh | inline probe | target (host:port), timeout, optional username/password (secret references arrive with the driver spec) | the tcp pair, ssh-handshake-time, endpoint-responsive, endpoint-authenticated (only when a credential was tried) | | tls | inline probe | target (host:port), timeout, tls-mode (verify, pin or skip; derived from the driver’s tls-mode input, verify when unset) | the tcp pair, tls-handshake-time (one per probe the component answered), endpoint-responsive, and for a certificate the mode accepted tls-certificate-expiry and tls-certificate-days-left | | udp | the driver’s request rung | target (host:port), timeout, listen-port (derived from the driver’s listen-port input; unset, the node picks one) | endpoint-responsive (here the verdict too), udp-response-time (one per probe the component answered); never tcp-open. A bare udp endpoint has no request to send and lands nothing |

The raw-tcp row below is built in its driver form: the line protocol over tcp holds a session (#816) that polls, commands, and listeners share (#817). So is the telnet row (#1033): the same line protocol behind a telnet filter that strips option negotiation and refuses every option (DO is answered WONT, WILL is answered DONT, a literal 0xFF escaped), its login the driver’s login dialog run as the session’s auth step on establish and every recover; proven against the simulated panel’s nvp-telnet face (login, poll, command, push, recover with a fresh login, and a wrong password rejected after one attempt); while the node holds an endpoint’s telnet session and it reads up, the reachability probe takes its verdict from the session rather than dial a second connection (gear that admits one telnet client would refuse it). And so is ssh (#1032): the line protocol over a shell channel (with a PTY when the driver’s session.pty asks, or an exec channel, session.exec), its userauth the session’s auth step with the credential the driver names (session.credential, a username with a password or a private key, or keyboard-interactive answered by the login dialog). A refused credential is auth, and the host key is pinned on first use: the node trusts the first key it sees and publishes it as ssh-host-key, the server delivers it back as the pin on every worklist, and a changed key reads rejected, class host-key, naming both fingerprints, which the gate carries as data. An operator who trusts the new key (a replaced component) retries the endpoint with it (:retry with host_key, audited with both keys; the console’s Trust this key), which declares the new pin and redials onto it; a key the refusal on record does not name is refused. A declared pin is trusted only as that path wrote it (its row’s source is key-pin): a declared row any other path left is never delivered, and a pin the worklist stops delivering is one the node forgets, trusting on first use again (ADR-1100). Proven against the simulated panel’s ssh shell (poll, command, push, recover onto the same key, down after the budget, a wrong password and a rotated key each rejected) and end to end (ADR-0157).

tls

Section titled “tls”

Built (#1035, ADR-0160): tls is the line protocol over a TLS connection, its own transport rather than an option on tcp, since an endpoint is named by its transport (a component holds a raw control port and a TLS one side by side) and its rung is the handshake. The handshake is one primitive: https consumes it as its own registry entry by the same rule (http and https), and the websocket and mqtt carriers consume it later. Each connection, the held session’s and the probe’s, runs the TCP connect and then the handshake under the endpoint’s verification mode, its driver’s tls-mode input, edited in place like a host or a port:

  • verify checks the certificate against the node’s system roots, with the host input as the server name. Any failure (an unknown authority, a name mismatch, an expiry) is a certificate refusal carrying its reason, never a timeout: the gate’s endpoint-rejection-reason, held session or not, which the console’s refusal notice shows (#1110), beside the held session’s session-reason where one is held, and a node self-log, said once when the probe meets it. A driver that declares no mode verifies.
  • pin trusts the first public key the component offers and refuses any other, as the ssh host key is pinned. The pin is the key (its SubjectPublicKeyInfo, in the same SHA256: form as an ssh fingerprint), not the certificate, so a renewal that keeps the key is no change, and a certificate past its not-after still connects under its pinned key (its days left read negative, which a rule alarms on). The node publishes the key it trusted as tls-certificate, the server delivers it back as the pin on every worklist, and the node keeps one pin per endpoint, shared by its session and its probe and carried across a session’s recycle. A changed key reads rejected, class certificate, naming both keys, and an operator who trusts it (a replaced component) retries the endpoint with it (:retry with certificate, audited), which declares the new pin. The seeded newtron-nvp-tls pins by default, since the gear it speaks to self-signs.
  • skip checks nothing: an operator’s explicit choice, never a driver’s default (a spec defaulting to it is refused at write), shown on the endpoint’s row as unverified.

The probe lands the handshake rung’s timing (tls-handshake-time, one observation per probe the component answered) and, for a certificate the mode accepted, its not-after as the tls-certificate-expiry property and the days left as the tls-certificate-days-left metric, the number a threshold rule reads (an expression cannot read the clock). Under a certificate gate the probe connects only, so those freeze at the last accepted certificate; the reachability read withholds them while the gate holds rather than show them as current, and serves the pinned key in pin mode alone. A bare tls endpoint’s tls-mode param is checked where it is written, as a driver’s input is. TLS 1.2 is the floor, and the component is authenticated alone: no client certificates, and no per-endpoint CA bundle. Proven against the simulated panel’s nvp-tls face over a real handshake (poll, command, push, recovery onto the same key, a changed key held beneath the gate across a node restart and accepted with :retry, verify refusing the self-signed face as certificate, skip connecting) and end to end.

http and https

Section titled “http and https”

Built (#902, ADR-0902): http is a full collection transport, stateless: every poll and every command is a request of its own, with no connection held (a login’s session, below, is state the node keeps, never a held connection). https is the same over the shared TLS primitive, its own registry entry as tls is (ADR-0160’s rule: its rung is the handshake, and a component may serve a plain web port beside a TLS one).

The request. A driver’s http request declares its method (GET unless it says otherwise), path, query, headers, a body and the response’s format: json (the default), xml, html or text. The format is declared, never sniffed from the Content-Type, since gear mislabels its content. A command’s request renders its ${value} and ${arg.key} references injection-safe: a path substitution is escaped as one segment (a value cannot add a / or a ?), a query value is encoded, a header value carrying a control character is refused, a JSON body is rendered leaf by leaf and then encoded (a leaf that is exactly one reference keeps the value’s type, so an intended volume rides as a number), and an xml body’s substitutions are escaped as text. A request never sets the credential headers (Authorization, Cookie) or the ones the client owns, and never names a secret.

The answer, through one extraction primitive. An emit locates its value with exactly one selector, bound to the declared format at write: a jsonpath (the dotted path) on a json answer, an xpath on an xml answer (namespace-aware, its prefixes declared on the request’s namespaces) or on an html answer parsed as a browser would (an OEM component’s status page), and a regex on any body as text. A selector yields exactly one value: an element’s text (trimmed) or an attribute’s value, or a scalar expression’s result (count(...)); one that finds nothing, or more than one node, faults that emit by name while the rest still land. An answer is parsed once, however many emits read it.

Auth is the session block’s scheme. session.credential names the secret input and session.scheme says how it rides every request: basic (a basic-auth username and password), bearer (a bearer-token), api-key (an api-key, in a header or a query parameter as the driver’s session.api_key places and names it), or digest (a basic-auth username and password, below). The credential is written only into the outgoing request: no error, reason, label or log line carries it, a query key is masked in every URL the node names, and a component’s text is scrubbed of it before it reaches a reason.

The status table. A rule the driver declares terminal decides first (session.terminal may match a response’s status, its body, or both: a gear’s 403 for a wrong credential, an “account locked” body). Then:

| answer | reads as | |---|---| | 2xx | success: the body is the payload | | 401 to a credentialed request | a terminal auth refusal at the session rung (over a login session, a lapse first: below) | | 408, 429, 5xx | retryable: the poll backs off, never sooner than its Retry-After, never later than its interval | | 3xx | not followed: the request’s own fault, naming where it pointed (a redirect to a login session’s declared login page is a lapse) | | any other 4xx (a 403, a 404) | the request’s own fault: a collection-failed naming the status, the poll waiting its interval, nothing gated | | no answer (refused, reset, timed out) | retryable, at the transport rung | | a certificate the mode refuses | certificate at the transport rung, as tls |

A body past 1 MiB (64 KiB for a command’s reply) faults naming the cap.

The session rung is proven by the credentialed exchanges. A credentialed 2xx lands endpoint-authenticated yes, and a refusal no; a redirect, a 403, another 4xx or a 5xx never judged the credential, so proves nothing either way. A refusal gates the endpoint at the session rung for its connection fingerprint: no credentialed poll or command is sent beneath it, across a node restart too, until the fingerprint moves (the secret rotated, an input edited in place, an operator’s :retry). The gate publishes its reason beside it, endpoint-rejection-reason (auth: the component refused the credential (401)), so the console’s refusal notice says why and offers Retry on an endpoint holding no session (ADR-1110). The reachability probe sends no credential, so reachability stays observed beneath the gate. A probe request that carries the credential (for a driver with commands alone) is #1120.

https. The endpoint’s driver’s tls-mode input (verify, pin or skip) judges the certificate as it does for tls, and its public key is pinned on first use as tls-certificate, one pin per endpoint, shared by its probe, its polls and its commands; a changed key reads rejected, class certificate, naming both keys, and :retry with certificate trusts the offered one. The probe runs the handshake as its rung (tls-handshake-time, one per answered probe), lands the certificate’s expiry and days left for a certificate the mode accepted, then draws a response (http-response-time). Beneath a certificate gate only the TCP connect runs.

Commands. An http command’s delivery carries its rendered request and never a credential: the node applies the endpoint’s scheme from what its worklist delivered, so a credentialed http driver that declares commands needs a poll to carry its credential (refused at write otherwise). The answer is judged by the same table; its body (scrubbed of the credential) is the reply a binding settling from the response reads through the same extraction (settle.extract may be a jsonpath or an xpath, settle.success a match on the body).

The seeded kestrel-api reads the simulated Kestrel bar over https with basic auth (its system, call and audio polls; set-volume, set-mute and set-power), and http-json reads the simulated Tessellate sensor over http with its header key. Proven over real sockets (internal/collection/httpfetch_integration_test.go: the request and its key, a redirect not followed, the body cap, a query key never named, verify, pin and skip, the https probe) and end to end against the simulator (internal/node/run_http_test.go: the bar polled and actuated three ways, each command settled from its reply, a replaced key held beneath the gate across a restart until trusted; a wrong header key sent once per fingerprint across a restart, a rotated key bringing it up; a bearer token over namespaced XML and a query key over an HTML status page).

Logging in over http

Section titled “Logging in over http”

Built (#1114, ADR-1114): much gear’s web API has no per-request credential. It has a sign-in: a form post that sets a session cookie, or a JSON login that answers a token. An http session declares it as session.login, the same login dialog a line session runs (#1061), carried over to http: each step is a request rather than a prompt, and its references are only the secret fields it sends, ${secret.<input>.<field>}.

session:
  login:
    path: /login                       # a redirect back here means the session lapsed
    steps:
      - request:
          method: POST
          path: /login
          body_format: form
          body: { username: "${secret.login.username}", password: "${secret.login.password}" }

A JSON login that answers a token captures it through the one extraction primitive and names the headers that carry it on every later request, ${session.<capture>}; an expiry reads how long the session lasts (seconds), and expired is the text a request draws once it has lapsed:

session:
  login:
    steps:
      - request: { method: POST, path: /api/login, body: { username: "${secret.login.username}", password: "${secret.login.password}" } }
    capture: [{ name: token, extract: { jsonpath: token } }]
    expiry: { jsonpath: expires_in }
    headers: { Authorization: "Bearer ${session.token}" }
    expired: { regex: "token expired" }

The session lives in the node’s memory alone. Each http endpoint has one auth state on its node, shared by its polls and its commands: a cookie jar holding the component’s cookies, the values the login captured and the headers they fill, and a digest challenge. It is never logged, published or persisted, and it is dropped whole when the endpoint’s connection fingerprint moves (every login input is a fingerprint input: the steps, the captures, the headers, the paths, each secret’s sealed version), on a terminal refusal, and with the node process. A login runs on a fresh jar before the first request, and again before a declared expiry lapses (with a tenth of the lifetime to spare), never once per poll. A declared lifetime is held between 10 seconds and 30 days (gear says “never” with a huge number); one the answer omits, or gives as zero or a word, declares none, and the session lasts until a request finds it lapsed. Only a login session keeps cookies: a request under a scheme is its own, as before, and a jar holds at most 64 cookies of at most 4 KiB.

A lapse logs in once. A request over a login session whose answer is a 401, a redirect to the declared login path, or the declared expired text has lapsed: the node logs in once more and replays the request once. A lapse right after a fresh login is the credential’s terminal auth refusal (auth: the component refused the session a fresh login opened (401)), never a second login.

A refused login is one attempt. A login step’s answer is judged as a login: a rule the driver declares terminal, a 401, a redirect back to the login page, any other 4xx, or a 2xx at the last step without the declared ready text (or without a value it captures) is the component refusing the login, a terminal auth at the session rung (auth: the login was sent back to the login page (302 to /login)). It is held as refused for the life of the auth state, the gate holds every request beneath it across a restart, and the next login is the one the moved fingerprint brings. No retry loop can lock an account out. A login that draws no verdict after its secrets were sent (a 5xx, a 408, a 429, a connection dropped once the request was written) is retried on the poll’s backoff, but three in a row spend the login budget, as a line session’s does, and read rejected with the component’s last answer; a connection never made sends nothing and charges nothing. A redirect anywhere else passes (a form login’s way of saying yes).

Digest (RFC 7616) is scheme: digest on a basic-auth secret. The first request draws the component’s challenge, which the node answers (MD5, SHA-256, or their -sess forms; qop=auth, or the legacy form with none), then holds, answering every later request at once with a rising count. A challenge the component marks stale, or one under a nonce it rotated without saying so, is answered afresh, never read as a refusal; a 401 to an answer of the nonce it still offers is the credential’s refusal. A 401 offering no digest challenge the node answers (Basic alone, an algorithm other than MD5 or SHA-256) is the request’s own fault, its schemes named: the credential was never sent, so it says nothing about it. The password never crosses the wire.

Never logged. The values the auth state holds (the secret fields a login sent, the values it captured, the component’s cookies) are scrubbed from every reason and error as the credential is, in every form a component could echo them, and from every URL the node names (a login step may render a secret into its query, though never into its host: a step’s path opens with a literal /). What a login would capture is scrubbed even from the reason that refuses it. A captured value with a control character, or longer than 8 KiB, refuses rather than riding a header, and a command whose delivery names an address the node’s worklist does not is never sent, so a session opened with one component never rides a request to another. The node logs that it logged in and why (opening, renewing before expiry, the session lapsed), never a value.

Shaped for OAuth2. A client-credentials grant (#1119) is this shape with no new primitive: a request step posting the client’s secret fields, a captured access_token, its expires_in, and a bearer header.

The seeded newtron-panel-web reads the simulated Newtron panel’s https API behind its form login, and boreal-edge-web the simulated Boreal display’s web admin over digest (and drives its power). Proven over real sockets (internal/collection/httpsession_integration_test.go: one session across many requests, one re-login and replay on a lapse, a wrong password one attempt and never retried, a token renewed before its expiry, concurrent requests sharing one login, digest with MD5 and SHA-256 and a stale nonce, nothing the state holds in any reason) and end to end against the simulator (internal/node/run_http_login_test.go: a form and a token login keep one session across many polls and log in exactly once more when the component forgets it; digest answers its challenge, a stale nonce, and a command; a wrong password reads rejected with its reason after exactly one attempt, across a node restart, and the secret fixed in place brings each back).

A server-sent events stream

Section titled “A server-sent events stream”

Built (#1038, ADR-1038): many components’ web pages update live from a server-sent events stream (text/event-stream), so a driver can hold that stream and land each change the moment the page would, with no poll. The session block declares it, a GET held open, and listeners claim its events:

session:
  credential: login
  scheme: basic
  stream: { path: /api/events }          # format json (the default) or text
listeners:
  - name: volume-change
    match: { event: change, field: { path: path, equals: status.audio.volume } }
    emits:
      - { name: volume, extract: { jsonpath: value } }

The stream is a held session. A standing listen task holds it through the staged dialer every held session uses: the connect is the TCP connect (and the TLS handshake over https, its pin the endpoint’s), the auth step logs in first when the session declares a login, then sends the stream’s GET with the scheme’s credential, the login’s session and the component’s cookies, and reads the answer’s head, and only then does the reader start. A 200 carrying text/event-stream opens it; a refusal is judged as any request’s is (a 401 is a terminal auth, a lapsed login logs in once more, a digest challenge is answered), and anything else (a 404, an answer that is not a stream) is a failed attempt the session retries on its backoff. session-state and session-reason land as for a held line session, endpoint-authenticated lands when the stream carries a credential, and the registry’s http and https entries read held because they now are. The endpoint’s learned auth (the jar, a login’s session, the digest challenge) is the one its polls and commands share, so one login serves them all.

A message, not a line. The held connection’s unit is a message: a line on a line session, one event here (its data, its event name, its id). The router offers each to the listeners first, as it offers a line, and a listener’s match claims one by its event name (event, alone or with a data rule), a prefix or regex on its data, or a JSON field match (field: { path, equals } or { path, regex }, the dotted jsonpath grammar, compared as the extraction renders it). A claimed event is read through the one extraction primitive in the stream’s declared format (a jsonpath on json data, a regex on text). An event no listener claims follows the spec’s unmatched policy. A stream has no upstream, so its listeners declare no arm.

Resume. The event-stream interpretation is the WHATWG standard’s (event, data, id, retry, comments, every line terminator). A dropped stream reads recovering and reconnects with Last-Event-ID, the last event id it was sent (carried across a reconnect that saw no event), after the server’s retry (held between 250 ms and 30 s, the session backoff’s bounds). A server replays the events after that id, so nothing is lost and nothing comes twice; one that replays from that event, sending it again first, has that one event dropped. A component that reboots and numbers its events from 1 again has every new event land. The resume lives with the endpoint in the node’s memory, across a session’s recycle while the endpoint reaches the same address, never persisted. An event whose data passes 1 MiB is dropped whole (its id still counts, and it is named unclaimed) and the stream reads on; an id past 1 KiB is ignored; an answer’s head is held to 64 KiB, and to the endpoint’s timeout, as a refused answer’s body is. A component must send the stream’s head when it opens it: one that holds the head back until its first event never opens while quiet.

Beside polls. The endpoint’s polls and commands stay requests of their own. A refusal on the stream gates them through the one fingerprint gate; a refusal a poll or a command meets stops the stream too (it restarts rejected, and every reconnect checks the gate before it sends), and the stream coming up never lifts a gate one of them met; and while the stream makes its first attempt under a fingerprint, the endpoint’s polls and commands wait for its verdict rather than sending the same credential beside it. A poll’s own fault (a 404, a redirect) waits its interval, whatever the stream beside it reads.

The seeded kestrel-api holds the simulated Kestrel bar’s stream beside its polls: its call, volume, mute and power arrive the moment they change. Proven by the event-stream interpretation’s tests (internal/collection/sse_test.go), over real sockets (httpstream_integration_test.go: events in order, a drop resumed after the server’s retry with nothing lost or twice, a refused credential one attempt, digest answered across a reconnect, a form login run first and again on a lapse, an uncredentialed stream proving no credential, an answer that is not a stream never a refusal), in the simulator’s events face, and end to end (internal/node/run_http_stream_test.go: a change on the bar lands within a second with no poll; a dropped stream reads recovering and resumes, landing each change once; a wrong password reads rejected with its reason, sent once under the fingerprint, across a restart; the secret fixed in place brings it back; a stream behind a form login logs in first).

udp

Section titled “udp”

Built (#1037, ADR-0163): udp is the line protocol as datagrams, one line each, and stateless: there is no connection to hold and no session-state. Each udp endpoint binds its own socket on the node, on its driver’s listen-port input (unset, a port the kernel picks), and every request it makes leaves from that socket, so the component’s replies and its pushes arrive on the one socket with no table of sources to keep. The operator configures the component to send to the node’s address and the endpoint’s port (and opens the node’s firewall for it); a shared node port routed by source address was rejected, since two endpoints behind one address (the IP-to-serial gateway shape) would be indistinguishable. The listen port is derived into the endpoint’s params as host and port are, so editing it moves the connection fingerprint. A datagram from any source but the endpoint’s component is counted, self-logged once per tick (foreign datagrams dropped, with the count and the last source), and dropped, never published.

An open udp port cannot be detected, so the transport rung is the driver’s named request answered (ladder.transport.request, which a udp driver must name): an answer reads responsive, a silent component reads not responsive (timeout), and a port the host reports unreachable reads refused. The socket is not connected (a connected socket filters other sources in the kernel, and they must be counted), so the node turns on the socket’s error queue (IP_RECVERR) to hear that report; it names a cause and never gates. With no port to open, the responsive rung is the endpoint’s verdict too, and no tcp-open lands; the ping stays informational, so a component that blocks ping but answers reads up, and one that answers ping but not the request reads down. Each answered probe lands one udp-response-time, an error-shaped answer included, timed from the request’s first send, so a lost datagram’s retry shows as time (a rung-specific name until #1063 folds probe timings into rtt); loss shows through endpoint-responsive and its reason. The reachability read serves the rung as a request layer, timed by udp-response-time only while answered, and the Endpoints panel’s breakdown lists it beside the ping, and a ping-up endpoint whose request goes unanswered says so in its why-line.

An exchange takes its reply by the request’s declared shape (request.reply, which every udp request must declare), with the held line connection’s precedence, so a datagram that does not match is never the answer and a push arriving mid-exchange still reaches its listener; a reply matching the error shape fails it with the component’s text. A component slower than a send’s share answers every send, so an exchange whose answer came back too soon after its last send to be that send’s reply (sooner than half the latency the socket learned from sends answered at once) leaves surplus replies behind; until they arrive, or its timeout passes again, a repeat of its answer is absorbed, never taken as the next ask’s. A retry that loss caused leaves nothing, since a lost send draws no reply, so a next ask’s identical answer (a common ack) is never starved. A reply later than the whole timeout is not told apart, so the endpoint’s timeout should exceed the component’s latency. A lost datagram is re-sent: request.retries (udp only, 0 to 5) defaults to 2 for a poll and for the request rung, and the attempts share the endpoint’s one timeout, so loss never stretches a tick. A command is sent once unless its binding declares retries, since a command may be relative (a volume step) and a lost reply must not send it twice; the seeded newtron-nvp-udp declares them on set-input, which names the input. A reply the driver declares terminal (its session block, which over udp may declare only terminal replies) gates the endpoint at the transport rung, since nothing logs in over udp, and beneath the gate no request is sent under that fingerprint. Listeners ride the socket as they ride a held session: their instantiation commands are sent once each (a subscribe that adds a destination must not add one per retry) when the socket opens and again whenever the request rung recovers (a component that rebooted forgot its subscriber), and never beneath a gate. Text payloads only, unicast only. Proven over real sockets (internal/collection/udp_integration_test.go, internal/node/datagram_test.go) and end to end against the simulated panel’s nvp-udp face dropping 30% of datagrams each way (run_udp_test.go: the rung up through the loss and its timing, the poll through its retries, a command actuated, a push with no poll, a stranger’s datagram dropped, a silent face and a refused port down, no tcp-open and no session-state).

snmp

Section titled “snmp”

Built (#821, ADR-0821): snmp is v2c and stateless. Its transport rung is a get of a standard OID answered, ladder.transport.request.get, by default 1.3.6.1.2.1.1.3.0 (sysUpTime.0), so snmp-generic gains its rung with no spec change and a driver may name another (sysObjectID.0, say); a rung request that names a line, or an OID that is not a numeric one, is refused at write. The get carries the endpoint’s own community, the secret its driver’s community input references, which the worklist delivers to the reachability probe as it does to the polls, so a rotated community moves the fingerprint like any secret. Any response is an answer, a noSuchObject included, as any http status is: the rung reads responsive, lands one snmp-response-time (a rung-specific name until #1063 folds probe timings into rtt), and, since a v2c agent answers only a community it accepts, endpoint-authenticated yes. With no port to open, the rung decides the endpoint’s verdict, and no tcp-open lands; the ping stays informational, so an agent that blocks ping but answers reads up.

The v2c limit, stated plainly: a wrong community is dropped silently by the protocol (RFC 3584), so it reads not responsive, timed out, retryable, and cannot be told apart from a stopped agent. Only an agent that answers authorizationError (a community it knows and refuses) refuses detectably: a terminal auth refusal at the session rung, endpoint-authenticated no, and since v2c has no request without the community, nothing is sent under that fingerprint, neither the probe’s get nor a poll’s (a restarted node adopts the gate too), until the fingerprint moves (rotating the community secret brings it up within a tick). The get is snmp’s only transport probe, so beneath the gate the agent is unobserved: the rung reads down, rejected, and the verdict with it, rather than freezing at the last answer. An snmp driver must declare its community, a secret input named community of type snmp-community, which the poll and the rung both read. SNMP v3, whose report PDUs make auth detectable, is #1083. The read serves the rung as a request layer timed by snmp-response-time, and the Endpoints panel names it the get, its why-line naming the v2c limit instead of a control port. A v2c scalar get only: no walk, no tables, no v1. Proven against a BER agent of its own (snmp_integration_test.go), the simulated faces (strict, and answering a refused community with authorizationError), on the node over fakes, and end to end (run_snmp_test.go: the get answered and timed, the poll beside it, a wrong community silent and never rejected, the refused community gated and not sent again until the secret rotated, a stopped agent down), with loss injected through the impairments under NET_ADMIN.

The other held transports are each their own issue: #1034 (websocket), #1036 (mqtt), #1039 (modbus/tcp), #1040 (bacnet/ip), #1041 (knxnet/ip); a server-sent events stream is held over http and https (above). There is no serial transport: an RS-232 or RS-485 device is reached through an IP-to-serial gateway, which is the tcp transport (or tls or telnet), so its driver is a line protocol like any other (ADR-0153).

icmp/tcp are inline probes (config on endpoint.params); snmp is built as its own stateless transport, configured by its driver’s inputs (snmp), and http and https as request transports configured by their driver’s spec, a declared stream held as a session (http and https); the rest, in the design model, are held connections: the connection lives on the endpoint, the task names what to read. The ship-with registry seeds the probe built-ins above, the endpoint verdicts, and session-state (internal/seed/metric_types.yaml, internal/seed/property_types.yaml; the metrics and properties guides render each set from the seed); the wider per-transport set (<proto>-reachable/<proto>-response-time for the remaining text family; an http answer’s status is a poll fault’s reason, never a sample of its own; telnet-response-time, udp-response-time and snmp-response-time are seeded, and there is no udp-open, since an open udp port cannot be detected) is Design, and unregistered names would be reject-not-project dropped at ingest.

Listeners over a held session

Section titled “Listeners over a held session”

Built (#817): a listener declared in a driver’s spec derives a standing listen task, and the node runs every listen task on an endpoint over that endpoint’s one held session:

  • Instantiation commands (arm): the subscribe conversation, sent in order on every connection before the session reads up, so on establish and again on each recover. They are plumbing, not command rows (the #603 ruling): each is a node self-log (session armed, with the device’s reply). The ack is the first line of any kind, one a listener claims included (the listener has it too), since a device may acknowledge a subscribe with the push itself; an arm with no ack within the timeout is logged at warn (session armed, unacknowledged) and does not fail the attempt, while the connection dying during the arm does.
  • Match (prefix or regex): classifies an inbound line. Listeners are offered a line in function-name order and the first whose rule picks it claims it. The unit is a message (#1038): a line on a line session, an event on an http stream, which a match may also claim by its event name (event) or by a JSON field of its data (field: { path, equals | regex }) (a server-sent events stream).
  • Extraction: the claimed line runs through the same emit shape a poll’s answer does (locate, transform, type by the baked lane), and lands at once as a sample under the listen task: the component owns it, the endpoint is its instance, and no poll schedule is involved. A claimed line whose emits do not extract is a collection-failed, landed once per change.

Every inbound line on a held session is routed by one precedence (ADR-0151):

  1. A line a listener claims is the listener’s. It is never an exchange’s answer, unless the exchange declared a reply shape the line matches, in which case both have it.
  2. An exchange in flight (a poll, a command) takes the first line its declared reply shape (request.reply, a prefix or regex) matches, or, when it declares none, the first line no listener claims. When the shape also declares an error shape, an unclaimed line matching it answers the exchange as a refusal carrying the device’s text. A line that began before the request was written is never its answer.
  3. Anything left is unclaimed: counted, and once per tick the node logs the count and the last such line; a spec declaring unmatched: collection-failed also lands one collection-failed per tick naming them (the default is discard).

A request with a reply shape closes the window #816 left open (a stray line arriving just before the device reads the request); without one, a line no listener claims in that instant can still be taken as the answer, so a driver whose protocol mixes unsolicited lines with answers declares its replies. A shape’s regex is Go RE2 with substring semantics, so anchor it (^INPUT \S+$), or it also matches a push containing it. Because shape routing cannot hand a late answer to the next ask, a shaped request that gets no answer in time fails only that exchange: the connection is kept and the session is not charged (an unshaped one still recycles the connection, ADR-0150). A poll whose answer is a refusal lands a collection-failed naming the device’s text. Commands ride the held session too: a delivery names its endpoint, and a node holding a session there actuates over it (falling back to a throwaway connection only when none is held). A command the device answers settles from that answer as it did over a throwaway connection; one it refuses reports the device’s text as both its execution error and its reply, so it fails at once with the reason on the row rather than timing out.

Listeners and their config

Section titled “Listeners and their config”

Execution: parse at the edge

Section titled “Execution: parse at the edge”

A function runs the parse at the edge, not server-side:

  • Steps parse, extract, and normalize on the node, publishing resolved samples to the JetStream samples stream, not the typed tables; the persistence consumer batch-writes them async, idempotent on (series, ts), while the rule engine consumes the same stream live. The wire is per-lane, so the lane rides the message shape itself, routing metric versus property.
  • Raw payloads are not stored: a dev raw-mode taps the wire live, a failed parse emits a collection.failed event carrying the raw, and there is no telemetry table.
  • Owner attribution: a single-owner function stamps identity at the edge; a many-device function publishes for multiple owners, resolved server-side from identity labels (below).
  • Placement-scoped writes. A node publishes only the owners in its placement visible_set (the owners of its assigned tasks), expressed as NATS subject permissions (the node gateway mode, identity and access). An owner label outside the set is never an authoritative write, only an orphan / discovery candidate for the discovery_rule stream, so a compromised node cannot manufacture writes for owners it was never placed on; the perspectives / disagree model backstops a legitimately-placed node reporting bad values.
  • No separately authored transform rule: routing is the template’s fan-out, cross-entity rollups are calc samples on system and location templates; the server-side remainder is shared-endpoint owner-binding and untemplated raw ingress.

Raw sampling: an opt-in re-parsable window

Section titled “Raw sampling: an opt-in re-parsable window”

An opt-in raw_sample policy keeps a bounded window of raw frames so a corrected extractor can re-derive its samples over that window, without reintroducing a telemetry table. It is cascade-resolved (settable on an endpoint, task, or template) to off (default, no raw retained), all (every frame buffered), or 1-in-N (sampled, bounding a high-cadence source). Kept frames carry the immutable function version that parsed them, so the window re-parses against that exact version and corrects retroactively; outside it, a wrong-but-conforming parse is forward-fixable only, the raw being gone. A short-TTL holding pen, range-partitioned and cold-tierable (storage), not a parallel history of record.

Shared-API collection: one component, many owners

Section titled “Shared-API collection: one component, many owners”

Some sources describe many entities at once: a SaaS / UCC platform (Zoom, Teams), a controller fronting many devices, a building gateway. The API is one component (one endpoint, one credential) whose data fans out to the entities it describes, not one component per described entity.

  • Its function pulls the batch in one call, labeling each emitted sample with the external identity it belongs to (a Zoom Room ID).
  • The function is the conduit, not the owner: ownership resolves server-side against a declared identity config (zoom-room-id on the target), the same shared-ingress owner-binding used for webhooks and traps.
  • The owner can be a system. SaaS state that is telemetry of a room maps to system-owned samples; a virtual component is for the genuine member case (its own topology node, a health_role, a lifecycle). Member -> component, telemetry -> system.
  • Unmatched identities are orphans, discovery candidates (below).

Best practice: map SaaS / cloud telemetry to system-owned samples wired into system health via a system-scoped event_rule; the vendor’s own status is an input to that judgment, not the verdict, so corroborate a reported “offline” before downing the room (health).

Identity binding: the value-to-owner index

Section titled “Identity binding: the value-to-owner index”

Binding an externally-tagged row to an Omniglass owner is a lookup against the value-to-owner index: an identity arc on identity config, a (property_type, value) -> owner mapping where property_type is the match key (zoom-room-id) and value the emitted identity. The index resolves in the cascade scope the identity config is set at, so an identity declared at a system or location binds every member below it. A declared identity config value wins, falling back to the observed identity sample sharing the key; the samples ownership-resolution machinery reads this same index.

discovery_rule: orphans become candidates

Section titled “discovery_rule: orphans become candidates”

A discovery_rule turns the orphan stream, every emitted identity the index does not resolve (an unmatched shared-API (property_type, value), plus a node’s out-of-placement labels), into proposed entities: the onboarding win, no hand-mapping.

  • Creates candidate components or owners, each seeded with the identity that surfaced it (it becomes the new entity’s identity arc), so the next batch resolves through the index.
  • Idempotent on re-discovery: keyed on the (property_type, value) already bound, so a steady stream of the same orphan resolves to one candidate.
  • Scope and standing: a cascade scope and an official / private standing like the other rule families (event_rule, calc).

Drivers: the declarative menu

Section titled “Drivers: the declarative menu”

A driver turns a class of device API into a normalized menu of signals and actions. Its body is the spec: a versioned declarative document (data, not code) interpreted by one generic engine, stored as jsonb on the driver row and validated against the catalogs at every write (create, update, and the boot seed), so a spec that cannot be interpreted refuses at authoring time, never at collection time (ADR-0135). A spec declares:

  • The transport it rides (one, from the code registry): snmp-generic rides snmp, newtron-nvp rides tcp, kestrel-api rides https.
  • Inputs an attachment must supply: typed (string, number, secret), required or defaulted. A secret input carries a secret_type and is supplied as a reference (the name of a secret row of that shape), never a value; the reference is checked at attach.
  • Three function families, every emitted sample referencing its lane catalog by name:
    • Poll functions: a schedule, a transport-shaped request (scalar OIDs for an SNMP get, a line for a line protocol, a method, path, query, headers, body and response format for http, #902), and per-emit extraction, one primitive with a selector per format (oid, regex capture, jsonpath, key, xpath), with an optional declared transform (cast, scale, enum map; the expression engine is deliberately not a dependency).
    • Listeners: the arm conversation sent on session establish and re-sent on every recover (session plumbing, not recorded command rows), a match rule that claims an unsolicited inbound payload, and the same extraction shape. A listener rides a held line session or a udp socket; on any other transport it is refused at write, since nothing would run it (#1115).
    • Command bindings: how a command_type actuates over this protocol.

Attach is the authoring move: creating an endpoint with a driver reference (instead of a transport) derives everything from the spec. The transport and params come from the spec (the host and port inputs join into the probed target), required inputs must be supplied and defaults bake, and the spec’s functions derive the endpoint’s tasks in the same transaction: a poll task per poll function, a standing listen task per listener, each task’s spec carrying the function whole with its emit lanes resolved at attach (the compile step: the node never looks up a catalog at runtime). The tasks re-derive when the driver’s spec changes (#823): a driver update over the API, and the boot seed’s upsert of an official spec, rewrite every attached endpoint’s driver tasks in the same transaction, writing the ones the spec now derives and retiring the ones it no longer does. Task ids are content hashes, so an unchanged function keeps its row and a re-seed that changes nothing writes nothing at all (not the driver row, not a task); an endpoint whose tasks changed has its config generation moved, and its node runs the new spec from its next pull. An attach reads the driver row under a share lock, so one that races an update waits for it and derives the committed spec, and a re-derive skips an endpoint a concurrent delete removes. One gap is accepted: between the re-derive and the node’s next pull (one tick), samples the node still publishes under a retired task’s id are dropped at ingest as orphans, the same confinement that drops any unknown task id; the next tick publishes under the new one. Refusals name the fault as a 422: a missing required input, an input the spec does not declare, an ambiguous secret name. A secret input is resolved under the caller’s own secret read scope and admin tier, so an attach can only bind a credential the operator could themselves read: an out-of-scope or admin-sensitive secret refuses indistinguishably from an absent one, no name or type oracle. The worklist delivery re-confines to a secret that cascades onto the endpoint’s component. A driver-attached endpoint’s address is derived from its inputs and edits in place through them (#1061): PATCH /endpoints/{endpoint} takes inputs (the host, the port, an attached secret’s reference), validated against the driver’s declared inputs (a value it refuses is a 400 naming the field), every attached secret re-resolved under the editor’s own secret scope (so an edit cannot send a credential its editor could not use to a new address), audited, and the tasks re-derived in the same transaction through #823’s one derivation; a raw params.target on it, or a transport its driver does not speak, refuses. A driver row without a spec is a stub that cannot be attached. The console’s endpoint create offers both moves (a bare probe endpoint, or attach a driver and fill its inputs), and the Drivers page renders each spec as the menu it is.

The interpreter runs the poll functions (#814) on their declared schedule (#822): a poll’s schedule.every is honored at heartbeat granularity, the node running it on the first tick at or after it is due (a new task at once, and a run whose fetch failed retried on a backoff from the heartbeat, doubling, capped at the interval), while reachability probes run every tick and a listener’s pushes ride the held session as they arrive, never gated (ADR-0152). The node tells a baked driver task from a probe task by its spec, fetches over the transport (an http request rides the node’s http fetcher with the endpoint’s credential, above; the v2c scalar GET rides gosnmp, proven against a from-the-spec BER agent; a line poll runs one stateless ask over a throwaway connection, or borrows the endpoint’s held session when the node holds one), and interprets the payload against the baked function: locate each emit by its one extractor, apply the declared transform (enum map, then scale, then cast), and type it by the baked lane. A metric-lane value that is not numeric is a fault, never a wrong-lane sample; a missing emit faults by name while the rest of the payload still lands; a failed fetch lands nothing; every fault is a collection-failed event naming the task. Secret inputs reach the node unsealed in the worklist reply: the server resolves each reference once per pull (the per-node NATS grant is the isolation boundary, and the attach that bound the reference is the audited operator action), and a reference that no longer resolves is delivered as absence, which the node reports as the missing credential. Sessions and the listeners over them are built (above). The ship-with set:

NameVersionRidesMenu
snmp-generic1.0.0snmp1 poll
kestrel-api1.0.0https3 polls, 5 listeners, 3 commands
newtron-nvp1.0.0tcp1 poll, 1 listener, 1 command
newtron-nvp-telnet1.0.0telnet1 poll, 1 listener, 1 command
newtron-nvp-ssh1.0.0ssh1 poll, 1 listener, 1 command
newtron-nvp-tls1.0.0tls1 poll, 1 listener, 1 command
newtron-nvp-udp1.0.0udp1 poll, 1 listener, 1 command
http-json1.0.0http2 polls
newtron-panel-web1.0.0https1 poll
boreal-edge-web1.0.0http1 poll, 1 command

Storage

Section titled “Storage”

The connection registry, the declared connections, and the node’s units of work; the physical layout lives on storage.

The transports are a code registry the binary ships (internal/transport), not a table: ADR-0073 decided it and #811 executed it, retiring the transport table and its FK. The registry renders below from the same generated facts the seed doc uses, so this list cannot drift from what a create refuses:

NameDescription
icmpICMP echo / ping reachability and RTT.
tcpRaw TCP: the connect / port-open check, and the line protocol, which holds a session over it (polls, commands, and listeners share one connection).
tlsTLS: the line protocol over a TLS connection, held exactly as it is over tcp; the probe runs the handshake (responds) and reports the certificate's expiry, verified against the node's roots, pinned to the public key on first use, or (by the operator's explicit choice) not verified.
udpUDP: the line protocol as datagrams, one line each, stateless: the probe sends the driver's named request and an answer is the rung (an open udp port cannot be detected), a refused port reads refused, and a lost datagram is retried; each endpoint binds its own local port (its driver's listen-port input, else one the node picks), which its requests, replies and pushes share.
telnetTelnet: the line protocol over TCP with option negotiation (every option refused), its login the session's auth step; the probe connects and waits for the first negotiation or banner bytes (responds).
sshSSH: the probe runs a real key exchange (responds) and tries the endpoint credential when one is set (auth); the line protocol holds a shell or exec channel over it, its userauth the session's auth step and its host key pinned on first use.
httpHTTP: a driver's polls and commands send real requests (method, path, query, headers, body) and read JSON, XML, HTML or text answers, authenticated by the endpoint's declared scheme (basic, bearer, an API key, digest) or a login whose session the node keeps; a declared server-sent events stream is held as a session, its events landing through the listeners; the probe draws a real response off the API without a credential (responds), any status counting as an answer.
httpsHTTPS: http over a TLS connection, its own entry so a component can serve a plain web port beside a TLS one, a server-sent events stream held over it as over http; the probe runs the handshake (responds), reports the certificate's expiry, then draws a response, the certificate verified against the node's roots, pinned to the public key on first use, or (by the operator's explicit choice) not verified.
snmpSNMP v2c, stateless: the probe gets a standard OID (sysUpTime.0 unless the driver names another) with the endpoint's community, and an answer is the rung; a wrong community is dropped silently and reads as no answer, while an authorizationError reads as an auth refusal.

endpoint is the connection, the authored primitive (nodes): name derived from the endpoint type, a set component_id means pre-bound while NULL means shared/match-key, and placement cascades from the node so a node purge drops its endpoints.

endpoint (fleet): generated from the migrated schema by make gen
Column Type Constraints Notes
id uuid PK, default uuidv7()
name text not null
params jsonb not null, default '{}'::jsonb The transport parameters the authoring schema validated
created_at timestamp with time zone not null, default now()
updated_at timestamp with time zone not null, default now()
component uuid FK → component.id
node_name uuid FK → node.principal_id Placement: holds the node primary key (principal_id), not a name, despite the column name; ON DELETE CASCADE so a purged node takes its endpoints
label text The friendly string an operator reads, and the ONLY identity string they type here since the name is derived; NULL falls back to that name, verbatim
transport text not null
driver_id uuid FK → driver.id Set when a driver attach authored this endpoint (#813); NULL for a bare probe endpoint
inputs jsonb not null, default '{}'::jsonb The effective inputs supplied at attach, defaults baked; secret inputs are reference names, never values
session_retry bigint not null, default 0
CHECK constraints and unique indexes on endpoint
  • endpoint_component_name_key: CREATE UNIQUE INDEX endpoint_component_name_key ON public.endpoint USING btree (component, name)

task is a derived unit of collection work: created with its endpoint, read-only (no operator CRUD), its id a content hash. There is no node column; placement projects from the endpoint. Parsing to samples is the edge function, not the task’s job.

task (collection): generated from the migrated schema by make gen
Column Type Constraints Notes
id text PK The content hash of (endpoint, mode, spec); same inputs, same task
label text
mode text not null poll or listen
endpoint_id uuid FK → endpoint.id, not null
spec jsonb not null, default '{}'::jsonb What to collect; the edge function parses the result into samples
enabled boolean not null, default true
created_at timestamp with time zone not null, default now()
updated_at timestamp with time zone not null, default now()
CHECK constraints and unique indexes on task
  • task_mode_check: CHECK ((mode = ANY (ARRAY['poll'::text, 'listen'::text])))