Skip to content
Omniglass Omniglass Omniglass

Data collection

Partial

Interfaces: the device’s APIs, declared once

Section titled “Interfaces: the device’s APIs, declared once”

An interface is an API we intend to call on a component (its web, qrc, ttp, snmp endpoint), not a network interface; a host or IP is a variable it consumes, not its identity. It is named by the protocol it speaks, unique within its component. Reachability is the first gate of a ladder (reach, auth, responds, collecting), needing only the transport. The top-level interfaces array declares each once; steps reference one by id, the interface owns the connection, and a persistent session outlives any single function run, so subscriptions and inbound streams attach to a connection established once.

interfaces:
  - id: snmp
    type: snmp                     # interface_type registry entry (param schema)
    host: ${input.ip}              # references INPUTS, not $var: directly (see Inputs)
    version: "2c"
    auth: ${input.snmp}            # snmp_community shape; community field is secret (masked, audited)
    liveness: { oid: 1.3.6.1.2.1.1.3.0 }   # reachability gate, per interface
  - id: cli
    type: ssh
    host: ${input.ip}
    credentialRef: ${input.ssh}    # ssh_credential shape, bound to a $var: at apply
    persistent: true               # stateful session, outlives function runs
  • Type is the interface_type, which is the transport, not the protocol driver: the registry covers snmp, http, ssh, telnet, tcp, icmp, webhook, mqtt, syslog, and websocket, each with a connection-param schema and default reachability probe. The protocol handler turning a device’s API into a normalized menu of samples and functions is a separate driver layer, so a device’s OIDs live in its driver, never on a template (ADR-0039).
  • A label is the one identity string an operator types on an interface. The name is derived from the type, so every SSH interface in the fleet reads ssh: the row says which protocol it speaks and nothing about what it is for. The label (“Control processor”) is where that goes, and it is settable at create rather than on a following edit, an interface labelled by a second call being unlabelled at the moment it is made (ADR-0119). It is optional; an interface with none reads its derived name verbatim. Note the corollary of a derived name: since the name IS the protocol and is unique within the component, a component holds at most one interface per protocol, and a second of the same type is a 409.
  • liveness: the per-interface reachability gate deciding whether its functions run (nodes).
  • persistent: true keeps a session open across runs; scheduled functions borrow it to send, listen functions wake on its inbound.
  • Codec and framing: raw-TCP AV control planes wrap payloads (line terminators, length prefixes, NUL framing, JSON-RPC or TTP envelopes); the interface’s codec frames outbound and parses inbound to the declared envelope, so a step sees structure, not wire bytes.
  • Node placement is not declared here; it is server-assigned from the component’s location.

Two axes: task mode and interface transport

Section titled “Two axes: task mode and interface transport”

A task is a node’s unit of collection work, described by two independent axes:

  • Task mode (on the task): poll (we ask) or listen (we wait), stated from our perspective on purpose, since “pull/push” inverts with the frame.
  • Transport (on the interface): stateless (a throwaway connection per shot) or stateful (a held-open connection, a session emitting session_log rows for connect/auth/drop/reconnect).

All four cells are real:

| | poll (we ask) | listen (we wait) | |---|---|---| | stateless | SNMP get, HTTP GET | webhook, SNMP trap, syslog | | stateful | SSH-exec or xAPI xStatus on a held session | MQTT subscribe, xAPI feedback |

Waiting for a frame is one mode (listen) regardless of transport; statefulness lives on the interface.

Native push: data pushed by smart senders is self-describing (it carries its key), its edge parse a near-identity pass-through marked shape=native; a failed parse keeps the raw on a collection.failed event.

Interface types and their config

Section titled “Interface types and their config”

The poll types and listeners in the interface_type registry and the operator config they read. How the node executes these (tick scheduling, reachability gating, the task queue) is nodes.

Poll protocols and their config

Section titled “Poll protocols and their config”

Built today: the icmp and tcp probes (with ssh and http probing as tcp-connect). A probe’s config rides the interface’s params JSON, unmarshalled on the node:

| interface type | shape | config (interface.params) | samples | |---|---|---|---| | icmp | inline probe | target (host or IP), count, timeout | icmp-reachable, icmp-rtt-avg (fixed) | | tcp | inline probe | target (host:port), timeout | tcp-open, tcp-connect-time (fixed) |

icmp/tcp are inline probes (config on interface.params); the rest, in the design model, are held connections: the connection lives on the interface, the task names what to read. The ship-with registry seeds the 11 real catalog rows, 4 metric_type plus 7 property_type (internal/seed/metric_types.yaml, internal/seed/property_types.yaml), including the probe built-ins above plus interface-reachable; the wider per-transport set (udp-open, snmp-reachable, the http trio, <proto>-reachable/<proto>-response-time for the text family) is Design, and unregistered names would be reject-not-project dropped at ingest.

Listeners and their config

Section titled “Listeners and their config”

Execution: parse at the edge

Section titled “Execution: parse at the edge”

A function runs the parse at the edge, not server-side:

  • Steps parse, extract, and normalize on the node, publishing resolved samples to the JetStream samples stream, not the typed tables; the persistence consumer batch-writes them async, idempotent on (series, ts), while the rule engine consumes the same stream live. The wire is per-lane, so the lane rides the message shape itself, routing metric versus property.
  • Raw payloads are not stored: a dev raw-mode taps the wire live, a failed parse emits a collection.failed event carrying the raw, and there is no telemetry table.
  • Owner attribution: a single-owner function stamps identity at the edge; a many-device function publishes for multiple owners, resolved server-side from identity labels (below).
  • Placement-scoped writes. A node publishes only the owners in its placement visible_set (the owners of its assigned tasks), expressed as NATS subject permissions (the node gateway mode, identity and access). An owner label outside the set is never an authoritative write, only an orphan / discovery candidate for the discovery_rule stream, so a compromised node cannot manufacture writes for owners it was never placed on; the perspectives / disagree model backstops a legitimately-placed node reporting bad values.
  • No separately authored transform rule: routing is the template’s fan-out, cross-entity rollups are calc samples on system and location templates; the server-side remainder is shared-interface owner-binding and untemplated raw ingress.

Raw sampling: an opt-in re-parsable window

Section titled “Raw sampling: an opt-in re-parsable window”

An opt-in raw_sample policy keeps a bounded window of raw frames so a corrected extractor can re-derive its samples over that window, without reintroducing a telemetry table. It is cascade-resolved (settable on an interface, task, or template) to off (default, no raw retained), all (every frame buffered), or 1-in-N (sampled, bounding a high-cadence source). Kept frames carry the immutable function version that parsed them, so the window re-parses against that exact version and corrects retroactively; outside it, a wrong-but-conforming parse is forward-fixable only, the raw being gone. A short-TTL holding pen, range-partitioned and cold-tierable (storage), not a parallel history of record.

Shared-API collection: one component, many owners

Section titled “Shared-API collection: one component, many owners”

Some sources describe many entities at once: a SaaS / UCC platform (Zoom, Teams), a controller fronting many devices, a building gateway. The API is one component (one interface, one credential) whose data fans out to the entities it describes, not one component per described entity.

  • Its function pulls the batch in one call, labeling each emitted sample with the external identity it belongs to (a Zoom Room ID).
  • The function is the conduit, not the owner: ownership resolves server-side against a declared identity config (zoom-room-id on the target), the same shared-ingress owner-binding used for webhooks and traps.
  • The owner can be a system. SaaS state that is telemetry of a room maps to system-owned samples; a virtual component is for the genuine member case (its own topology node, a health_role, a lifecycle). Member -> component, telemetry -> system.
  • Unmatched identities are orphans, discovery candidates (below).

Best practice: map SaaS / cloud telemetry to system-owned samples wired into system health via a system-scoped event_rule; the vendor’s own status is an input to that judgment, not the verdict, so corroborate a reported “offline” before downing the room (health).

Identity binding: the value-to-owner index

Section titled “Identity binding: the value-to-owner index”

Binding an externally-tagged row to an Omniglass owner is a lookup against the value-to-owner index: an identity arc on identity config, a (property_type, value) -> owner mapping where property_type is the match key (zoom-room-id) and value the emitted identity. The index resolves in the cascade scope the identity config is set at, so an identity declared at a system or location binds every member below it. A declared identity config value wins, falling back to the observed identity sample sharing the key; the samples ownership-resolution machinery reads this same index.

discovery_rule: orphans become candidates

Section titled “discovery_rule: orphans become candidates”

A discovery_rule turns the orphan stream, every emitted identity the index does not resolve (an unmatched shared-API (property_type, value), plus a node’s out-of-placement labels), into proposed entities: the onboarding win, no hand-mapping.

  • Creates candidate components or owners, each seeded with the identity that surfaced it (it becomes the new entity’s identity arc), so the next batch resolves through the index.
  • Idempotent on re-discovery: keyed on the (property_type, value) already bound, so a steady stream of the same orphan resolves to one candidate.
  • Scope and standing: a cascade scope and an official / private standing like the other rule families (event_rule, calc).

Storage

Section titled “Storage”

The connection registry, the declared connections, and the node’s units of work; the physical layout lives on storage.

interface_type is the transport registry, decided to become a code registry (one package per transport), retiring this table and its FK (ADR-0073); still a table today.

interface_type (fleet): generated from the migrated schema by make gen
Column Type Constraints Notes
name text not null
official boolean not null, default false
description text not null, default ''::text
built boolean not null, default false Whether the transport ships in the binary today
created_at timestamp with time zone not null, default now()
id uuid PK, default uuidv7()
CHECK constraints and unique indexes on interface_type
  • interface_type_name_key: CREATE UNIQUE INDEX interface_type_name_key ON public.interface_type USING btree (name)

interface is the connection, the authored primitive (nodes): name derived from the interface type, a set component_id means pre-bound while NULL means shared/match-key, and placement cascades from the node so a node purge drops its interfaces.

interface (fleet): generated from the migrated schema by make gen
Column Type Constraints Notes
id uuid PK, default uuidv7()
name text not null
params jsonb not null, default '{}'::jsonb The transport parameters the authoring schema validated
created_at timestamp with time zone not null, default now()
updated_at timestamp with time zone not null, default now()
component uuid FK → component.id
node_name uuid FK → node.principal_id Placement: holds the node primary key (principal_id), not a name, despite the column name; ON DELETE CASCADE so a purged node takes its interfaces
type uuid FK → interface_type.id, not null
label text The friendly string an operator reads, and the ONLY identity string they type here since the name is derived; NULL falls back to that name, verbatim
CHECK constraints and unique indexes on interface
  • interface_component_name_key: CREATE UNIQUE INDEX interface_component_name_key ON public.interface USING btree (component, name)

task is a derived unit of collection work: created with its interface, read-only (no operator CRUD), its id a content hash. There is no node column; placement projects from the interface. Parsing to samples is the edge function, not the task’s job.

task (collection): generated from the migrated schema by make gen
Column Type Constraints Notes
id text PK The content hash of (interface, mode, spec); same inputs, same task
label text
mode text not null poll or listen
interface_id uuid FK → interface.id, not null
spec jsonb not null, default '{}'::jsonb What to collect; the edge function parses the result into samples
enabled boolean not null, default true
created_at timestamp with time zone not null, default now()
updated_at timestamp with time zone not null, default now()
CHECK constraints and unique indexes on task
  • task_mode_check: CHECK ((mode = ANY (ARRAY['poll'::text, 'listen'::text])))