Alarms and actions

Partial

An alarm detects a condition and holds it; an action does something about it. A simple action is a single step (a notify); a multi-step action is a flow. Both are stateful entities that hold their state directly (not event-sourced). An action’s credentials live in credentials; the Expr and Go-template machinery in expressions.

Built today: the alarm as a held row, raised on a component by a caller, carrying a severity (info / warning / critical), a message, a raised_at, and a nullable cleared_at (clearing keeps the row). Any active alarm reaches a system through the component’s own verdict (any active alarm degrades it, a critical one is an outage), and a critical alarm impairs every role that component occupies while it stays open, since only an outage component stops occupying its slots; a lesser alarm degrades the component but leaves it in place (#626: an alarm no longer names what it degrades).

Also built: acknowledgement (below), the record that a human has seen an incident, orthogonal to whether it is still raised. Snooze and resolve are not built, and the reasons are in that section. See implementation status.

The alarm (a stateful entity)

Metaphor: a morning alarm, not a pop-up. A pop-up alert appears and is gone (a notify action); an alarm goes off when its condition is met and stays high until interacted with. The alarm row holds its current state directly (status, opened_at, resolved_at in the target design; built today: severity, message, raised_at, cleared_at, acknowledged_at, acknowledged_by).

Target design (ADR-0050)

One incident, a new row per open, keyed by (event_rule, owner) (the exclusive-arc owner, so a system- or location-owned sample yields a system/location-owned alarm), the ITSM correlation anchor (samples). The open and resolve events carry the alarm_id and are the edge log; the alarm row is the live state, never reconstructed from them.

PG-first: the firing event_rule consumer writes the alarm row in the same Postgres transaction as the event, serialized per (event_rule, owner) so an incident never double-opens; a CDC publisher fans the committed transition onto JetStream for action_rule consumers. No dual-write; Postgres is never a message bus.

opened / resolved are rule-driven (fire_criteria / clear_criteria), each emitting an event with the alarm_id; the operator-driven transitions are recorded in audit_log (also carrying the alarm_id) and applied to the row. The timeline assembles by alarm_id across events + audit. The acknowledgement half of that is built and works exactly this way (below); snooze is not.

An alarm write is not sample-silent: raising or clearing one recomputes health in the same transaction, and the rollup records the verdict transition as a calculated-provenance property sample (provenance='calculated', source_rule='health-rollup'). Cycle safety therefore rests on provenance, not topology (ADR-0069, see Cycle safety).

Acknowledgement: a fact about a person

An alarm’s raised state is a fact about its condition: the one-open invariant is per (component, dedup_key) (ADR-0075), and it is the condition, not a person, that decides whether the alarm is still up. An acknowledgement is a fact about a person: somebody has seen this. The two are therefore orthogonal, stored as two nullable columns rather than one status enum, and an alarm sits in any of these states:

| | unacknowledged | acknowledged | | --- | --- | --- | | raised | nobody has looked; the queue an operator works | somebody is on it, and it is still broken | | cleared | it came and went unattended | it was seen, then fixed |

POST /components/{name}/alarms/{id}:acknowledge records it, gated by alarm:acknowledge (held by operator and above). Three properties follow from acknowledgement being a fact about a person:

It does not recompute health. Acknowledging is not fixing. A recompute here would record a verdict transition at a moment when nothing about the fleet changed, and the health history is meant to be edges and only edges.
It carries its own permission and its own scope, resolved on the component tier from alarm:acknowledge itself rather than from component:update. Recording that somebody looked is not editing the component, and a responder role may hold one without the other (identity and access).
It is idempotent. The first sighting is the one that matters operationally (time to acknowledge), so a second acknowledgement keeps the first person and the first time and writes no second audit row, the same shape as a re-raise of an open condition returning the existing incident.

Un-acknowledging is not a verb. If it turns out to matter, it is a second verb and a second decision, not a nullable field somebody flips (#728).

Both are commonly bundled with acknowledge, and neither is built, for reasons rather than sequencing:

Snooze exists to suppress notification. The outbound notification registry is unbuilt (#618), so a snooze today would suppress nothing: a column that lies. It belongs with #618 or after it.
Resolve is either the existing DELETE /components/{name}/alarms/{id} (clear) under a second name, or a different concept nobody has specified. A human “resolving” an alarm whose raised state belongs to its condition is a contradiction until somebody says what it means.

A three-verb permission string bundling acknowledge with both of them appeared for most of a year in this repo’s test fixtures and in an earlier draft of the identity and access page. It was never a design: nothing seeded it and no route ever enforced it (ADR-0109). The permission lint now catches any attempt to write it back, because alarm is a resource the route surface knows and neither of the other two verbs gates anything.

The event_rule (ADR-0050)

The `event_rule`

An event_rule carries a required fire_criteria and an optional clear_criteria. With a clear_criteria the fire event opens an alarm and the clear event resolves it; without one the rule is momentary (a one-shot event, no alarm). There is no separate alarm_rule.

event_rule:
  scope: 'component.template == "polaris-dsp-16"'   # the shared selector (expressions)
  sample: dsp.temperature
  window: { reduce: avg, over: 10m }                 # machinery (optional)
  fire_criteria: "value > 65"                        # opens the alarm
  clear_criteria: "value < 60"                       # resolves it (defaults to !fire_criteria)
  for: 0                                             # fire-side sustain (optional)
  for_clear: 0                                       # clear-side sustain (optional)
  severity: average

scope selects the entities (fan-out, one alarm per match); sample is the input; window / for / for_clear are aggregation machinery; fire_criteria / clear_criteria are the Expr leaves; severity names a level by id (below). A rule is suppressible by name through the cascade (cascade): a high-weight group removes a false-firing rule without editing it.

for and for_clear are symmetric sustains on the two edges, defaulting to 0, set independently: a long for_clear over a short for holds an incident through a noisy recovery without delaying the page.

A rule does not declare a health verdict itself: the alarm it raises impairs its owner’s own verdict (any active alarm degrades it, a critical one is an outage). Only an outage owner stops occupying the system roles it fills; a lesser alarm degrades it but leaves it in place. When it does drop out (#626), how much that matters is the role’s impact (health). A rule scoped to whichever arc owns its sample yields component-level and system-level alarms with one machinery (a system-scoped rule reads member data, fires a system-owned alarm).

The severity-level registry (ADR-0050)

Severity: a registry of named levels

Severity is a registry of named levels, not a bare integer. It takes the ordinary identity triad, so it adds no fourth vocabulary: a name on the entity name rule, an optional label, and its uuid id like any other row, plus a color and an integer order used only for comparison. Rules reference a level by name; official defaults ship spaced, so a new level inserts between two others without renumbering:

| name | label | color | order | |---|---|---|---| | info | Info | gray | 10 | | warning | Warning | yellow | 20 | | average | Average | orange | 30 | | high | High | red | 40 | | disaster | Disaster | dark red | 50 |

Severity is distinct from health (health); higher order is more severe. The set is operator-customizable (relabel to P1/P2/P3, recolor, insert, three levels or seven), all config. Predicates compare by level through the order (alarm.severity >= "high" matches high and disaster); the UI renders the label and color.

The action_rule (ADR-0050)

The `action_rule` (decoupled subscription)

Detection and response are separate (the event_rule does not contain its response, avoiding Zabbix’s action/operation tangle). An action_rule is a NATS consumer on the CDC-published event / alarm stream, selecting with an Expr predicate, so one rule serves many alarms. Subscriptions are indexed by event key and label, so dispatch evaluates only predicates whose key or label matches; a rule may carry multiple triggers and fires if any matches (including a label or wildcard trigger, e.g. any event labeled room=boardroom-a). It is a subscription, not a fourth sample-pipeline rule family:

action_rule:
  on: alarm
  when: 'transition == "open" && alarm.severity >= "high" && component.type == "device"'
  action: pagerduty-notify

The source is polymorphic but guarded (see Cycle safety): an alarm transition, a scheduled fire (an event with origin=scheduled), an operator (manual), and the declarative runbook step-list. Bodies are Go templates, sink auth a credential reference (expressions).

Storm and dependency suppression (ADR-0050)

Storm and dependency suppression

The alarm grain stays (event_rule, owner); two primitives keep the fan-out from becoming a page storm. Dependency suppression mutes a child alarm whose owner’s parent entity on the exclusive-arc structural tree is in an outage health state (open, not dispatched); the arc gives the parent walk for free. Action-level grouping coalesces alarms sharing owner / label / correlation_id into one action dispatch (one ticket with N members) at the dispatch edge; the alarms stay distinct rows.

A system-scoped event_rule is the sanctioned upstream-cause dedup lever. Worked example: a switch reboot downs 20 endpoints; the system-scoped rule owns the room-level cause as one system-owned alarm, dependency suppression mutes the 20 child alarms, and grouping coalesces the rest into one dispatch: the cause, not 20 symptoms.

Action durability and egress (ADR-0050)

Durability and egress

Action state is PG-first + CDC-out: the step transition commits to the action row, the leader-elected CDC publisher fans it onto JetStream, and the external send is a JetStream consumer (retry with backoff, dead-letter), not a Postgres SELECT ... FOR UPDATE SKIP LOCKED relay. Sends are at-least-once; sinks tolerate dupes or an idempotency key (alarm + action + transition) makes the outcome exactly-once. Pipeline order: render the body, then apply auth over the rendered bytes (HMAC signs the rendered body), then send. Egress safety is always on: block internal / metadata IPs, verify TLS, bound timeouts, control redirects.

Cycle safety in the action layer (ADR-0069)

Cycle safety in the action layer

The collection -> sample -> alarm core is acyclic by construction, and only data authors events: an event_rule over samples (plus the clock’s origin=scheduled) is the only way an event enters the log; flows and actions never manufacture events. That leaves one possible loop, the data-mediated control loop (an action commands a device, the new state arrives as a sample, opens an alarm, fires the action again), closed with three rules:

Consequence writes are provenance-labeled, and a rule never routes on its own consequences. Every consequence write carries provenance='calculated' with its producing rule named (the health rollup’s property row, source_rule='health-rollup'), and the routing layer refuses to trigger a rule off its own output (ADR-0069).
Operator transitions do not match action_rules (only open / resolve do). Acknowledgement is the built one and behaves this way already: it fires nothing, because it recomputes nothing. Snooze remains unbuilt and refused (ADR-0109).
The control loop is lineage-guarded at dispatch: before running an action, the engine walks the triggering event’s causation lineage (carried on NATS message headers); if the same (action, owner) appears upstream it is suppressed, with a depth bound as backstop. Flows are finite by construction (a step list, per-open-alarm, gated on open, cancelled on resolve or acknowledgement).

The walk crosses the device round trip because the command stamps its correlation_id onto the intended write and onto the adaptive-poll’s observed sample (samples), and the event_rule firing off that sample inherits the id. The carrier crosses the lane boundary as header (data lane) -> PG column -> header (record lane): the event_rule writes the sample’s correlation_id and a source_event_id parent edge onto the event row (events), and the CDC publisher re-emits both.

Correlation id: the trace of a causal chain

The same lineage powers a read-side trace: a correlation id threading the whole causal chain (sample -> event -> alarm -> flow / action -> command -> the value change that clears the alarm), stamped at the head and propagated along each caused edge on NATS message headers, so an operator can query “everything this one event set in motion” (the alarm_id links only one alarm’s own open / clear events). Not a sample kind, not a stored span subsystem: no new tables, no tracing backend.

Flows (ADR-0050)

Flows: the multi-step action

A flow is a bounded multi-step action: a DAG of steps (notify, command, wait, branch, parallel) over one alarm, instantiated per open alarm, gated on the alarm staying open, cancelled on resolve or ack, cycle-guarded by the lineage walk above, finite by a depth / step cap, its wait steps on the durable per-incident timer (time).

The canonical case is an escalation: remediate, wait, re-check the real sample the alarm is built on, escalate if unchanged. The same engine covers a time-bound access grant and an AI-troubleshooting flow. It is deliberately a bounded workflow engine, not Turing-complete; a flow does not author events, and any effect it has returns as ordinary data (the only edge that could re-open its alarm, lineage-bounded). A drag-and-drop editor edits flows. Single-step and multi-step are one model: only the length of the step list distinguishes them.

Storage

alarm and action are stateful entities that hold their current state directly (not event-sourced); the physical layout (the owner arc, partitioning) lives on storage.

The built alarm row is component-local: no owner arc and no event_rule. It does carry its acknowledgement (#728), as two nullable columns rather than a status enum, because an enum would force acknowledged and cleared into one column and make them mutually exclusive, which they are not. The target design shape adds event_rule, the owner arc, status, opened_at, and resolved_at.

`alarm` (telemetry): generated from the migrated schema by `make gen`
Column	Type	Constraints	Notes
`id`	uuid	PK, default uuidv7()
`severity`	text	not null	How bad; the rollup and the console sort on it
`message`	text	not null, default ''::text
`raised_at`	timestamp with time zone	not null, default now()	When the incident opened
`cleared_at`	timestamp with time zone		NULL while open; the one-open index keys on it
`created_at`	timestamp with time zone	not null, default now()
`updated_at`	timestamp with time zone	not null, default now()
`component_id`	uuid	FK → component.id, not null	The component the incident is about (the thin-cut owner)
`dedup_key`	text	not null	The condition identity (ADR-0075): one open alarm per (component, dedup_key), enforced by the partial unique index while open
`acknowledged_at`	timestamp with time zone		When a human first recorded seeing it; NULL while nobody has. Orthogonal to cleared_at
`acknowledged_by`	uuid	FK → principal.id	Who recorded it; ON DELETE SET NULL, so purging a principal keeps the time, and audit_log keeps the name

CHECK constraints and unique indexes on alarm

alarm_severity_check: CHECK ((severity = ANY (ARRAY['info'::text, 'warning'::text, 'critical'::text])))
alarm_open_condition_key: CREATE UNIQUE INDEX alarm_open_condition_key ON public.alarm USING btree (component_id, dedup_key) WHERE (cleared_at IS NULL)

A command is two real tables: the command_type registry and the command invocation row with its computed settlement; see commands. The event_rule / action_rule config rows live with the rule families.

Model-keyed command cascade

An abstract action (reboot) resolves to different concrete commands by component type / model through the cascade, so one reboot targets a heterogeneous fleet and each device runs the command its model declares; the edge dispatch and the command declaration live in templates / nodes.