Alarms and actions

An alarm detects a condition and holds it; an action does something about it. A simple action is a single step (a notify); a multi-step action is a flow. Both are stateful entities that hold their state directly (not event-sourced). An action’s credentials live in credentials; the Expr and Go-template machinery in expressions.

Partial

Built today: the alarm as a held row, raised on a component by a caller or by an event rule (#819), carrying a severity (a level of the severity registry, by name), a message, a raised_at, and a nullable cleared_at (clearing keeps the row). Any active alarm reaches a system through the component’s own verdict (any active alarm degrades it, one at or above the high level is an outage), and such an alarm impairs every role that component occupies while it stays open, since only an outage component stops occupying its slots; a lesser alarm degrades the component but leaves it in place (#626: an alarm no longer names what it degrades).

Also built: the event rule that raises and clears alarms on its own as samples land, with sustains held by a durable streak and a sweep, one official rule (reachability-lost), each rule’s live state, and the console’s Rules and Severity pages. The event-to-alarm edge (#938, ADR-0145): a rule’s alarm is optional (an informational rule raises none), it writes typed events at each edge of its condition, carrying the alarm they opened or resolved, and an alarm’s timeline merges those events with its audit rows. Acknowledgement (below), the record that a human has seen an incident, orthogonal to whether it is still raised. Snooze and resolve are not built, and the reasons are in that section. See implementation status.

The alarm (a stateful entity)

Metaphor: a morning alarm, not a pop-up. A pop-up alert appears and is gone (a notify action); an alarm goes off when its condition is met and stays high until interacted with. The alarm row holds its current state directly (status, opened_at, resolved_at in the target design; built today: severity, message, raised_at, cleared_at, acknowledged_at, acknowledged_by).

Target design (ADR-0050)

One incident, a new row per open, keyed by (event_rule, owner) (the exclusive-arc owner, so a system- or location-owned sample yields a system/location-owned alarm), the ITSM correlation anchor (samples). The open and resolve events carry the alarm_id and are the edge log; the alarm row is the live state, never reconstructed from them.

PG-first: the firing event_rule consumer writes the alarm row in the same Postgres transaction as the event, serialized per (event_rule, owner) so an incident never double-opens; a CDC publisher fans the committed transition onto JetStream for action_rule consumers. No dual-write; Postgres is never a message bus.

opened / resolved are rule-driven (fire_criteria / clear_criteria), each emitting an event with the alarm_id; the operator-driven transitions are recorded in audit_log (also carrying the alarm_id) and applied to the row. The timeline assembles by alarm_id across events + audit. The acknowledgement half of that is built and works exactly this way (below), as are the rule-driven events and the timeline itself (below); snooze is not.

An alarm write is not sample-silent: raising or clearing one recomputes health in the same transaction, and the rollup records the verdict transition as a calculated-provenance property sample (provenance='calculated', source_rule='health-rollup'). Cycle safety therefore rests on provenance, not topology (ADR-0069, see Cycle safety).

The alarm’s timeline

An alarm’s history is one read (#950): GET /components/{name}/alarms/{id}:timeline (omniglass component alarm timeline <name> <id>, gated alarm:read, the component resolved in the caller’s component read scope first, so a component it cannot read is the same non-disclosing 404 an unknown one is) merges the events that carry the alarm (event.alarm_id: an event rule’s fire event, which opened it, and its clear event, which resolved it) with its audit rows (the raise, each acknowledgement, the clear, and a rule edit that turned a severity on or off, naming the editor) into one list, oldest first. Each entry says when, whether it is an event or an audit row, what (the event’s type, or raise / acknowledge / clear), and who: the person who acted, or the rule the entry belongs to (the rule wrote the event, or the platform acted for it). At one instant an event comes before the audit row it caused, since a rule’s event and the alarm write it moves share the transaction’s clock. It is two bounded statements over the component the route already resolved (the alarm, then both sources in one ordered union over event.alarm_id and a partial audit_log index on alarm rows), never a read per entry. The id may come in any form Postgres reads as a uuid (uppercase, braced) and is matched in its canonical form. The platform’s audit lines name the rule as its events recorded it, so a deleted rule still names its lines. An open timeline refreshes with the alarm list, and an acknowledge or clear refreshes it at once. Hand-raised alarms write no events, so their timeline is their audit rows alone. In the console every alarm row carries a clock button that opens its timeline in place, under the row: the Why strip on a component’s and a system’s page and blade, the component’s alarms panel (active and recently cleared), and a rule’s open alarms.

Acknowledgement: a fact about a person

An alarm’s raised state is a fact about its condition: the one-open invariant is per (component, dedup_key) (ADR-0075), and it is the condition, not a person, that decides whether the alarm is still up. An acknowledgement is a fact about a person: somebody has seen this. The two are therefore orthogonal, stored as two nullable columns rather than one status enum, and an alarm sits in any of these states:

| | unacknowledged | acknowledged | | --- | --- | --- | | raised | nobody has looked; the queue an operator works | somebody is on it, and it is still broken | | cleared | it came and went unattended | it was seen, then fixed |

POST /components/{name}/alarms/{id}:acknowledge records it, gated by alarm:acknowledge (held by operator and above). Three properties follow from acknowledgement being a fact about a person:

• It does not recompute health. Acknowledging is not fixing. A recompute here would record a verdict transition at a moment when nothing about the fleet changed, and the health history is meant to be edges and only edges.
• It carries its own permission and its own scope, resolved on the component tier from alarm:acknowledge itself rather than from component:update. Recording that somebody looked is not editing the component, and a responder role may hold one without the other (identity and access).
• It is idempotent. The first sighting is the one that matters operationally (time to acknowledge), so a second acknowledgement keeps the first person and the first time and writes no second audit row, the same shape as a re-raise of an open condition returning the existing incident.

Un-acknowledging is not a verb. If it turns out to matter, it is a second verb and a second decision, not a nullable field somebody flips (#728).

Snooze and resolve are deliberately absent

Both are commonly bundled with acknowledge, and neither is built, for reasons rather than sequencing:

• Snooze exists to suppress notification. The outbound notification registry is unbuilt (#618), so a snooze today would suppress nothing: a column that lies. It belongs with #618 or after it.
• Resolve is either the existing DELETE /components/{name}/alarms/{id} (clear) under a second name, or a different concept nobody has specified. A human “resolving” an alarm whose raised state belongs to its condition is a contradiction until somebody says what it means.

A three-verb permission string bundling acknowledge with both of them appeared for most of a year in this repo’s test fixtures and in an earlier draft of the identity and access page. It was never a design: nothing seeded it and no route ever enforced it (ADR-0109). The permission lint now catches any attempt to write it back, because alarm is a resource the route surface knows and neither of the other two verbs gates anything.

The event_rule

An event_rule raises alarms on its own (#918, #919; ADR-0144). It reads one sample key (a registered property or metric, whose lane it records), carries a required fire_criteria and an optional clear_criteria, and names a severity level. When a landed sample meets the fire criteria the rule opens an alarm on the sample’s component; when one meets the clear criteria it clears it; a rule with no clear criteria clears on the negation of its fire criteria. There is no separate alarm_rule.

The alarm is optional (#948): a rule with no severity is informational and raises no alarm. A rule also names the event types it records: a required fire_event_type, written when its condition starts, and an optional clear_event_type, written when it ends, both by name from the event_type registry (an unknown one is refused, and a type a rule names cannot be deleted). The platform seeds a generic pair for rules that need no semantic type of their own, condition-started and condition-ended; a rule can name a bracketing pair the registry already models instead (call-started / call-ended).

Edges, not samples (#949). A rule’s condition is in force on a component and instance while it holds a row in event_rule_condition, for every rule, informational or not. When the condition comes into force (after fire_sustain) the rule writes the condition row, opens its alarm when it has a severity, and writes its fire event; when it leaves force (after clear_sustain) it deletes the row, resolves the alarm, and writes its clear event when it has a clear type. Each is written once, never per sample while the condition holds, in one transaction with the alarm transition and the health recompute. The event is origin = derived, names its rule (derived_by_rule_id, and the rule’s name and version in source_rule, the name it had when the event was written, which a rename leaves as written and which outlives a deleted rule), sits on the series’ component and instance, carries the rule’s message and the sample’s key and value as attributes, and carries the alarm it opened or resolved in event.alarm_id. A rule with no clear type resolves its alarm with no clear event, and its condition still ends. Turning a severity on opens alarms for the conditions in force, and turning it off resolves them while the conditions stay in force; neither writes a condition event, since the condition did not move, so an episode carries exactly one started and one ended event, and a rule edit’s alarm open or resolve is recorded in the alarm’s audit rows, naming the editor. On a material edit that also turns a severity on, the rule primes first, so a condition the edited rule no longer holds ends before an alarm could be raised for it.

A hand clear of a rule’s alarm holds until the condition ends and returns. In force is the condition row, not the alarm, so an operator clearing the alarm leaves the condition row and its events untouched, and a further sample of the same condition (a metric over its threshold on every poll included) does not reopen it; the next alarm opens on the next new edge, after the condition ends and starts again (the same manual close a Zabbix trigger allows); the one exception is a severity turned off and back on, which raises for every condition still in force, a hand-cleared one included. A condition in force is what the rule’s live counts and state read too, so open alarms never outnumber conditions. An alarm the rule holds open with no condition row behind it (one an older binary raised during a rolling upgrade) also counts as in force: a further fire neither adopts it nor writes an event, and a recovery resolves it with no clear event, since no condition row ended.

A rule’s alarm is found by its rule and instance, whatever key wrote it (#977). Every path that reads a rule’s alarms (the in-force check, the live counts and state, the reconcile pass, a recovery, and a disable, delete, narrowing or severity turned off) matches an open alarm by raised_by_rule_id plus the instance its key names, the suffix after the key’s rule segment. The rule segment may be the rule’s uuid, which this binary always writes, or its name, which a replica from before #977 still writes during a rolling upgrade after the migration ran (rule:<name>[:<instance>], the name it read then, renamed since or not). So such an alarm counts as in force (a further fire opens no second alarm and writes no event), is listed on its instance, and is resolved on recovery (by the clear event, when it has a condition row) or when the rule lets go of it. Only within one rule: a key’s rule segment never matches another rule’s alarm, since the match starts from raised_by_rule_id, and a hand-raised alarm may not take a key beginning rule: at all (a 422; the key defaults to the message, so a message beginning rule: needs a key of its own).

event_rule:
  name: reach-lost
  sample_key: endpoint-reachable       # a property (text) or a metric (number)
  fire_criteria: { expr: 'value == "down"' }
  clear_criteria: { expr: 'value == "up"' } # optional: absent, the negation of fire, !(fire)
  fire_sustain: 30s                     # how long fire must hold
  clear_sustain: 0s
  severity: high                        # optional: absent, informational (no alarm)
  fire_event_type: condition-started    # the event written when the condition starts
  clear_event_type: condition-ended     # optional: the event written when it ends
  selector: { component_types: [display], components: [] }   # empty: every component

The evaluator judges each sample as it lands, in the same transaction, through the same guarded conditional insert, audit row and health recompute a hand-raised alarm uses, so a rule-raised alarm feeds health exactly as a hand-raised one does; it writes as the system actor. The alarm names its rule (raised_by_rule_id, set null if the rule is deleted), and its condition identity (its dedup_key, rule:<uuid>[:<instance>]) is the rule’s uuid plus the sample’s instance, so a per-endpoint input holds one alarm per endpoint: a component whose SNMP and HTTP both fail carries two, and one recovering clears only its own. A repeated or redelivered sample changes nothing. A disabled rule evaluates nothing; disabling or deleting a rule ends the conditions it holds in force, each with its clear event, resolving their alarms (they stay in the history), and narrowing its selector ends those on the components it no longer watches.

A rule’s name is renameable; its references are by uuid (#977, ADR-0062, ADR-0076). POST /event-rules/{name}:rename moves an operator rule’s name, the address an operator types, and answers the rule with its uuid. Everything inside the platform that refers to a rule holds the uuid: an alarm’s raised_by_rule_id and its dedup key, the condition rows, the streaks, an event’s derived_by_rule_id, the audit rows. So a rename moves nothing else: the alarms it holds stay open and attached, its conditions stay in force, a streak keeps its clock, and the next recovery clears the alarm and writes the clear event under the new name. A rename is not a material change and bumps no version. An event’s source_rule is a historical record, the name the rule had when it wrote the event, so an episode that spans a rename reads the old name on its fire event and the new one on its clear, both by the same derived_by_rule_id. An official rule keeps its name (a 409), since the boot seed matches it by name. A one-time data migration rewrote the dedup key of every rule-raised alarm, open or cleared, from the rule’s name to its uuid; an alarm whose rule was deleted keeps the key it had.

Sustains hold an edge until it has lasted: fire_sustain before the alarm opens and clear_sustain before it clears, both defaulting to 0s, set independently (a long clear sustain over a short fire sustain holds an incident through a noisy recovery without delaying the page). A sample pointing at an action starts or continues a streak (stored per rule, component and instance, so it survives a restart); a sample that points nowhere (neither criterion holds), or at an action already in force (a fire with the condition in force, a clear with none), breaks it, as does a sample the rule cannot evaluate; and once it has held for the sustain the action takes effect, either on a later sample or on the server’s sweep (every five seconds, beside the command settlement sweep), which is what lets a sustain work on an input that reports only on change, like endpoint-reachable. Before acting, the sweep re-judges the series’ newest sample and checks the rule still selects the component, and it serializes with an arriving sample on the same series, so a recovery landing as the sustain runs out never leaves the alarm open. A rule write serializes with the samples judged by the rule too: an ingest holds the rule for its batch, so a disable, delete or narrowing waits for an in-flight batch and then clears whatever it raised. Sustains are timed by the server’s receipt of each sample, never by the sample’s own timestamp, so a node whose clock runs behind neither skips nor stretches a sustain (which sample is newest is still decided by that timestamp). Only a material edit restarts a rule’s streaks (its key, criteria, clear criteria, selector or sustains); a label, description, message, severity or event type edit leaves them running, since the event types change what the rule writes at an edge, not when the edge falls. Creating a rule, enabling one, or a material edit primes it from each selected series’ newest sample, so a condition that already holds starts its sustain then (or fires at once with no sustain) instead of waiting for a change the input may never send.

Two narrow races can still cost one decision on an input that reports only on change: a rule enabled (or re-keyed) while an ingest batch carrying its key is uncommitted primes from committed samples only, so the batch’s sample is judged by neither side; and a judgment Postgres aborts on a deadlock rolls back its savepoint, so the sample lands and the rule does not act. A reconcile pass heals both (#928): once a minute, on a loop of its own beside the sweep (so a slow pass never delays a sustain or a settlement) and under a deadline, the server reads every series that took a sample since the pass before last, re-judges each enabled rule’s newest sample against the alarm and streak it holds, and corrects only what disagrees (fires, clears, or starts the streak), each corrected series in a transaction of its own under the same lock order as an ingest (the rule’s row, the series lock, the health locks). Agreement is the common case and costs no write: the series, their newest samples (an index probe per series on either lane), the open alarms and the streaks are read in one statement per lane and judged in memory, and a streak that agrees keeps its clock. A restarted server’s first pass reads back a bounded number of recent samples, not the whole history, since the race it could have lost was in flight around its own start; a series whose correction fails is retried on its own for a few passes and then dropped with an error logged, while the pass moves on. The platform ships one official rule, rendered here from the seed itself:

Rule	Reads	Fires	Clears	Writes	Severity
reachability-lost (Reachability lost)	endpoint-reachable	value == "down" for 15s	value == "up"	condition-started, condition-ended	high

An endpoint whose verdict reads down for the sustain opens a high alarm on its component, and reading up clears it, so unplugging a simulated component takes its system out and plugging it back recovers it, with no operator in between. That takes the system out only when the component fills a role whose impact is outage and losing it drops the role below quorum; otherwise the system reads whatever the role’s impact says, and a component that fills no role moves no system.

Criteria are expressions (#957), stored as {"expr": "..."} and evaluated against a prepared input of five bindings: value (a metric’s number, or a property’s value typed by its data_type: text for string and, for now, json, a true or false for bool), key (the property or metric the rule reads), instance (the sample’s endpoint, empty when it has none) and component, the sample’s component, with its name and its type (the component type, by name). So value > 80 && instance != "aux" or value == "down" && component.type == "display". A criterion must yield true or false; one that names anything else, or yields another type, is refused when the rule is written (a 422 naming the criteria, line and column) and never stored. The criteria site admits a small vetted set of builtins and runs under a small budget (the expressions page lists both). A rule with no clear criteria clears on the negation of its fire criteria, !(fire).

The short form a rule could only be written in before, one comparison {op, value} (a property compares text with eq or ne, a metric compares numbers with any of eq ne gt ge lt le), is still accepted on input and lowered to its expression (eq "down" is value == "down", ge 80 is value >= 80.0), and the API always answers with the expression. A one-time backfill rewrote every stored short form to the same text the gateway’s lowering writes, and such a rule decides every sample as it did before with one deliberate exception: a bool property’s sample that is not exactly true or false now holds the rule, where the short form compared its text (the definition’s decision on #524). A sample the criteria cannot read (that bool case, or a number that is no number), or a run over its budget, holds the rule for that sample and is logged once per rule version per process; the sample still lands.

A rule’s criteria compile once per rule version, never per sample, and a compile that fails is held the same way. Criteria that stop compiling (a key’s data_type changed under them, or a short form the backfill left as it was because its expression would pass the engine’s source cap) hold the rule, and the rule says so: its list entry, a get and its state carry criteria_error (which criteria, the line and column, the message, and the diagnostic’s code when a registered one refused them), and the console shows the rule as one whose criteria do not compile until an operator rewrites them. Criteria that compile with a warning (value == 21.5, which a measured number seldom equals exactly) carry it in criteria_warnings: a note on the rule’s blade that never holds the rule or refuses its save.

Before a rule is saved it can be previewed (POST /event-rules:preview, gated rule:read, #959): the candidate (its key, selector and criteria) is judged over the last 24 hours of its key’s samples, on the components the caller may read, as the ingest would have judged them with no sustains (the answer says so): each series starts from the state its newest deciding sample in the week before set, and a late sample older than one before it decides nothing. The answer is the edges it would have written and the series it would hold in force now, and a series with no deciding sample in that week is reported as having started unknown rather than guessed at. It writes nothing. Its reads are index probes per selected series, capped at a row count taken a whole series at a time: a preview that hits the cap says how many series it judged of how many, flagged truncated. The console authors criteria either way: the rule drawer’s comparison builder shows the expression it is stored as, Write as an expression opens an expression field checked as it is typed (diagnostics under their line and column, the site’s bindings and functions beneath), and its Preview runs the draft; the rule’s blade shows its criteria as code (#960). The choices behind the engine and its first consumer are ADR-0146. The selector picks the components a rule watches: those whose type falls within a listed component type, and those listed by id; empty, every component. A rule outlives what it points at: deleting a component or a key it reads leaves it editable, and it simply matches nothing.

Rules read like a registry: any rule:read grant lists and gets every rule, wherever it is scoped. Writing one is fleet-wide: authored at /event-rules with rule:* on an all-scope grant, since a rule’s selector reaches every component (ADR-0144). A version bumps on every change.

A rule’s live state is a read, not a stored column: GET /event-rules/{name}/state (omniglass event-rule state list <rule>, gated rule:read) returns the conditions it holds in force (component, endpoint instance, since when; #951), the alarms it raised that are still open (component, endpoint instance, severity, when) and its streaks waiting out a sustain (component, instance, whether it points at fire or clear, since when). The list and a get carry the same state as counts (conditions_in_force, open_alarms, pending_streaks), aggregated for the whole list in one query, so a page of rules polls one list rather than a state read per rule. The state names components, so the lists and the counts cover only the components the caller’s component read scope admits: a reader scoped to one place sees every rule, but only the alarms and streaks it could already see. The console’s Catalog > Actions > Rules page (#921) lists every rule with its criteria in words, its severity and those live counts, opens a rule’s blade with both lists linked to their components, and authors, edits, disables and deletes operator rules in a drawer (official rules are read-only); the operator’s walk-through is the rules guide.

The live state also says when the rule last judged a sample (last_judged_at on the list, a get and the state route; absent when it never has), which tells a quiet rule from one whose key has stopped arriving (#929). A per-sample write was kept off the ingest path on purpose, so the stamp is coalesced: an ingest batch that judged a sample by a rule stamps it once, as its last statement, and only when the rule’s stamp is at least 5 seconds old; a stamp another batch holds is skipped rather than waited on. The time is honest to within those 5 seconds across every server replica, costs a rule reading a busy key one small write every 5 seconds, and lives in its own event_rule_live row, never on the rule row every batch holds. Only the judging of an arriving sample counts: a disabled rule, a component the rule does not watch, a sample older than its series’ newest, or one the rule cannot evaluate stamps nothing. It names no component, so it is not fenced by the read scope.

A rule does not declare a health verdict itself: the alarm it raises impairs its owner’s own verdict (any active alarm degrades it, one at or above high is an outage). Only an outage owner stops occupying the system roles it fills; a lesser alarm degrades it but leaves it in place. When it does drop out (#626), how much that matters is the role’s impact (health).

The rest of the event rule (ADR-0050)

Still design: an expression selector (scope: 'component.template == "polaris-dsp-16"', fan-out, one alarm per match) once the engine lands (#524); a window (window: { reduce: avg, over: 10m }) aggregating the input before the criteria; a rule suppressible by name through the cascade (cascade), so a high-weight group removes a false-firing rule without editing it; and a rule scoped to whichever arc owns its sample, yielding system-level alarms with the same machinery (a system-scoped rule reads member data and fires a system-owned alarm).

Severity: a registry of named levels

Severity is a registry of named levels, not a bare integer. It takes the ordinary identity triad, so it adds no fourth vocabulary: a name on the entity name rule, an optional label, and its uuid id like any other row, plus a color and an integer order used only for comparison. Rules reference a level by name; official defaults ship spaced, so a new level inserts between two others without renumbering. The shipped levels, rendered from the seed:

Level	Color	Rank	An active alarm at it
info (Info)	gray	10	degrades its component
warning (Warning)	yellow	20	degrades its component
average (Average)	orange	30	degrades its component
high (High)	red	40	puts its component in outage
disaster (Disaster)	darkred	50	puts its component in outage

Severity is distinct from health (health); higher order is more severe. The set is operator-customizable (relabel to P1/P2/P3, recolor, insert, three levels or seven), all config. Predicates compare by level through the order (alarm.severity >= "high" matches high and disaster); the UI renders the label and color.

Built (#917): the severity_level table (the order column is rank, since order is a reserved word), the five official levels above from the boot seed, and operator levels at /severity-levels (omniglass severity-level create, gated severity_level:*, a write on an all-scope grant; a name or rank already taken is a 409, an official level is read-only). An alarm’s severity references a level by name (a rename carries its alarms; a level an alarm names cannot be deleted). A component’s own verdict reads the rank: high and above is an outage, anything lower degrades. The retired three values mapped by meaning: critical became high, info and warning kept their names. The console reads the registry everywhere a severity shows: every alarm chip (the component’s alarms, a system’s and a place’s “why”, a rule’s open alarms) renders the level’s label in its color and sorts by its rank, so an operator level renders like a shipped one with no console change, and Catalog > Actions > Severity (#921) lists the levels worst first with their rank, color and effect, and authors operator levels in a drawer (#930): create, rename, relabel, recolor, re-rank and delete, gated severity_level:* on an all-scope grant like a rule write (ADR-0144, ruled 2026-10-20): a level’s rank moves health fleet-wide, so a holder whose grant stops at one place reads the levels but writes none (a 403). Official levels are read-only. Its form teaches the rank as it is typed: where the rank sits among the levels, and whether an alarm at it would take its component out or degrade it.

Open question

Whether a severity level is purely a label, color, and order, or also carries policy such as a default ack-timeout per level.

The action entity (ADR-0050)

The action (a stateful entity)

What an action_rule raises and runs; stateful, holding its own state directly:

• kinds: notify (in-app), webhook, email, run (execute a command; edge realization in templates / nodes);
• a simple action carries delivery state (queued / sent / failed / retried, the at-least-once JetStream consumer);
• a multistep action is a flow, carrying workflow state (current step, waiting, branches).

The action row carries identity, kind, config, and current status.

The action_rule (ADR-0050)

The action_rule (decoupled subscription)

Detection and response are separate (the event_rule does not contain its response, avoiding Zabbix’s action/operation tangle). An action_rule is a NATS consumer on the CDC-published event / alarm stream, selecting with an Expr predicate, so one rule serves many alarms. Subscriptions are indexed by event key and label, so dispatch evaluates only predicates whose key or label matches; a rule may carry multiple triggers and fires if any matches (including a label or wildcard trigger, e.g. any event labeled room=boardroom-a). It is a subscription, not a fourth sample-pipeline rule family:

action_rule:
  on: alarm
  when: 'transition == "open" && alarm.severity >= "high" && component.type == "device"'
  action: pagerduty-notify

The source is polymorphic but guarded (see Cycle safety): an alarm transition, a scheduled fire (an event with origin=scheduled), an operator (manual), and the declarative runbook step-list. Bodies are Go templates, sink auth a credential reference (expressions).

Storm and dependency suppression (ADR-0050)

Storm and dependency suppression

The alarm grain stays (event_rule, owner); two primitives keep the fan-out from becoming a page storm. Dependency suppression mutes a child alarm whose owner’s parent entity on the exclusive-arc structural tree is in an outage health state (open, not dispatched); the arc gives the parent walk for free. Action-level grouping coalesces alarms sharing owner / label / correlation_id into one action dispatch (one ticket with N members) at the dispatch edge; the alarms stay distinct rows.

A system-scoped event_rule is the sanctioned upstream-cause dedup lever. Worked example: a switch reboot downs 20 endpoints; the system-scoped rule owns the room-level cause as one system-owned alarm, dependency suppression mutes the 20 child alarms, and grouping coalesces the rest into one dispatch: the cause, not 20 symptoms.

Action durability and egress (ADR-0050)

Durability and egress

Action state is PG-first + CDC-out: the step transition commits to the action row, the leader-elected CDC publisher fans it onto JetStream, and the external send is a JetStream consumer (retry with backoff, dead-letter), not a Postgres SELECT ... FOR UPDATE SKIP LOCKED relay. Sends are at-least-once; sinks tolerate dupes or an idempotency key (alarm + action + transition) makes the outcome exactly-once. Pipeline order: render the body, then apply auth over the rendered bytes (HMAC signs the rendered body), then send. Egress safety is always on: block internal / metadata IPs, verify TLS, bound timeouts, control redirects.

Open question

The dead-letter surface and the operator retry of failed actions.

Open question

The observed-use auth-failure feedback from actions into credential health (paired with the credential-health model in config and credentials).

Cycle safety in the action layer (ADR-0069)

Cycle safety in the action layer

The collection -> sample -> alarm core is acyclic by construction, and only data authors events: an event_rule over samples (plus the clock’s origin=scheduled) is the only way an event enters the log; flows and actions never manufacture events. That leaves one possible loop, the data-mediated control loop (an action commands a device, the new state arrives as a sample, opens an alarm, fires the action again), closed with three rules:

1. Consequence writes are provenance-labeled, and a rule never routes on its own consequences. Every consequence write carries provenance='calculated' with its producing rule named (the health rollup’s property row, source_rule='health-rollup'), and the routing layer refuses to trigger a rule off its own output (ADR-0069).
2. Operator transitions do not match action_rules (only open / resolve do). Acknowledgement is the built one and behaves this way already: it fires nothing, because it recomputes nothing. Snooze remains unbuilt and refused (ADR-0109).
3. The control loop is lineage-guarded at dispatch: before running an action, the engine walks the triggering event’s causation lineage (carried on NATS message headers); if the same (action, owner) appears upstream it is suppressed, with a depth bound as backstop. Flows are finite by construction (a step list, per-open-alarm, gated on open, cancelled on resolve or acknowledgement).

The walk crosses the device round trip because the command stamps its correlation_id onto the intended write and onto the adaptive-poll’s observed sample (samples), and the event_rule firing off that sample inherits the id. The carrier crosses the lane boundary as header (data lane) -> PG column -> header (record lane): the event_rule writes the sample’s correlation_id and a source_event_id parent edge onto the event row (events), and the CDC publisher re-emits both.

Correlation id: the trace of a causal chain

The same lineage powers a read-side trace: a correlation id threading the whole causal chain (sample -> event -> alarm -> flow / action -> command -> the value change that clears the alarm), stamped at the head and propagated along each caused edge on NATS message headers, so an operator can query “everything this one event set in motion” (the alarm_id links only one alarm’s own open / clear events). Not a sample kind, not a stored span subsystem: no new tables, no tracing backend.

Flows (ADR-0050)

Flows: the multi-step action

A flow is a bounded multi-step action: a DAG of steps (notify, command, wait, branch, parallel) over one alarm, instantiated per open alarm, gated on the alarm staying open, cancelled on resolve or ack, cycle-guarded by the lineage walk above, finite by a depth / step cap, its wait steps on the durable per-incident timer (time).

The canonical case is an escalation: remediate, wait, re-check the real sample the alarm is built on, escalate if unchanged. The same engine covers a time-bound access grant and an AI-troubleshooting flow. It is deliberately a bounded workflow engine, not Turing-complete; a flow does not author events, and any effect it has returns as ordinary data (the only edge that could re-open its alarm, lineage-bounded). A drag-and-drop editor edits flows. Single-step and multi-step are one model: only the length of the step list distinguishes them.

Namespacing

Event rules and severity levels carry the same official boolean discipline as the rest of the registries (built, #917 and #920), seeded through each registry’s authoritative upsert (UpsertEventRule, UpsertSeverityLevel, beside UpsertPropertyType and siblings): official: true ships vetted and is read-only, official: false is operator-authored. An official level never takes over an operator’s: a clash stops the boot naming it.

Actions under the same discipline (ADR-0050)

Actions take the same official column when they land (#618), and operator rows are central to component templates.

Storage

alarm and action are stateful entities that hold their current state directly (not event-sourced); the physical layout (the owner arc, partitioning) lives on storage.

The built alarm row is component-local: no owner arc. It names the rule that raised it, if any, in a nullable raised_by_rule_id (set null when the rule is deleted), and it carries its acknowledgement (#728), as two nullable columns rather than a status enum, because an enum would force acknowledged and cleared into one column and make them mutually exclusive, which they are not. The target design shape adds the owner arc, status, opened_at, and resolved_at.

alarm (telemetry): generated from the migrated schema by make gen

Column	Type	Constraints	Notes
id	uuid	PK, default uuidv7()
severity	text	FK → severity_level.name, not null	A severity level by name (FK, ON UPDATE CASCADE); the rollup and the console sort on its rank
message	text	not null, default ''::text
raised_at	timestamp with time zone	not null, default now()	When the incident opened
cleared_at	timestamp with time zone		NULL while open; the one-open index keys on it
created_at	timestamp with time zone	not null, default now()
updated_at	timestamp with time zone	not null, default now()
component_id	uuid	FK → component.id, not null	The component the incident is about (the thin-cut owner)
dedup_key	text	not null	The condition identity (ADR-0075): one open alarm per (component, dedup_key), enforced by the partial unique index while open
acknowledged_at	timestamp with time zone		When a human first recorded seeing it; NULL while nobody has. Orthogonal to cleared_at
acknowledged_by	uuid	FK → principal.id	Who recorded it; ON DELETE SET NULL, so purging a principal keeps the time, and audit_log keeps the name
raised_by_rule_id	uuid	FK → event_rule.id	The event rule that raised it; NULL for a hand-raised alarm or once the rule is deleted

CHECK constraints and unique indexes on alarm

• alarm_open_condition_key: CREATE UNIQUE INDEX alarm_open_condition_key ON public.alarm USING btree (component_id, dedup_key) WHERE (cleared_at IS NULL)

The action entity (ADR-0050)

action is a stateful entity carrying delivery and step state (id, ordered steps of notify/command/wait/branch, status, current_step), driven by events and alarms. No action table exists today.

The event rule is three tables: the authored event_rule row, the durable streak each rule holds per component and instance while it waits out a sustain, event_rule_state, and the condition each rule holds in force per component and instance, event_rule_condition, which an upgrade backfilled from the rule-raised alarms open at the time. The severity registry is severity_level.

event_rule (telemetry): generated from the migrated schema by make gen

Column	Type	Constraints	Notes
id	uuid	PK, default uuidv7()
name	text	not null
label	text
description	text	not null, default ''::text
selector	jsonb	not null, default '{}'::jsonb	Component types and components it watches; empty, every component
sample_key	text	not null	The property or metric it reads, by name; sample_lane records which registry
sample_lane	text	not null
fire_criteria	jsonb	not null	The expression the condition starts on, {"expr": "..."} (#957); the short form {op, value} is lowered to it on input
clear_criteria	jsonb		The expression it ends on, {"expr": "..."}; NULL clears on the negation of fire_criteria
fire_sustain	interval	not null, default '00:00:00'::interval
clear_sustain	interval	not null, default '00:00:00'::interval
severity	text	FK → severity_level.name	The level its alarms are raised at, by name; NULL is an informational rule, which raises no alarm
message	text	not null, default ''::text
enabled	boolean	not null, default true
official	boolean	not null, default false
version	integer	not null, default 1	Bumps on every change
created_at	timestamp with time zone	not null, default now()
updated_at	timestamp with time zone	not null, default now()
fire_event_type_id	uuid	FK → event_type.id, not null	The event type written when the condition starts (FK event_type, required)
clear_event_type_id	uuid	FK → event_type.id	The event type written when the condition ends; NULL writes none

CHECK constraints and unique indexes on event_rule

• event_rule_clear_sustain_check: CHECK ((clear_sustain >= '00:00:00'::interval))
• event_rule_fire_sustain_check: CHECK ((fire_sustain >= '00:00:00'::interval))
• event_rule_sample_lane_check: CHECK ((sample_lane = ANY (ARRAY['property'::text, 'metric'::text])))
• event_rule_name_key: CREATE UNIQUE INDEX event_rule_name_key ON public.event_rule USING btree (name)

event_rule_state (telemetry): generated from the migrated schema by make gen

Column	Type	Constraints	Notes
rule_id	uuid	PK, FK → event_rule.id
component_id	uuid	PK, FK → component.id
instance	text	PK, default ''::text
pending	text	not null	fire or clear: the action the samples point at
since	timestamp with time zone	not null	The server receipt time the streak began; the sustain is measured from here

CHECK constraints and unique indexes on event_rule_state

• event_rule_state_pending_check: CHECK ((pending = ANY (ARRAY['fire'::text, 'clear'::text])))

event_rule_condition (telemetry): generated from the migrated schema by make gen

Column	Type	Constraints	Notes
rule_id	uuid	PK, FK → event_rule.id
component_id	uuid	PK, FK → component.id
instance	text	PK, default ''::text
since	timestamp with time zone	not null	When the condition came into force; one row while it holds, for every rule

severity_level (telemetry): generated from the migrated schema by make gen

Column	Type	Constraints	Notes
id	uuid	PK, default uuidv7()
name	text	not null
label	text
color	text	not null, default ''::text
rank	integer	not null	Higher is worse; at or above the health rule's outage floor an active alarm is an outage (the shipped levels, each marked, are on the rules guide)
official	boolean	not null, default false
created_at	timestamp with time zone	not null, default now()
updated_at	timestamp with time zone	not null, default now()

CHECK constraints and unique indexes on severity_level

• severity_level_rank_check: CHECK ((rank > 0))
• severity_level_name_key: CREATE UNIQUE INDEX severity_level_name_key ON public.severity_level USING btree (name)
• severity_level_rank_key: CREATE UNIQUE INDEX severity_level_rank_key ON public.severity_level USING btree (rank)

A command is two real tables: the command_type registry and the command invocation row with its computed settlement; see commands. The action_rule config row lives with the rule families.

Model-keyed command cascade

An abstract action (reboot) resolves to different concrete commands by component type / model through the cascade, so one reboot targets a heterogeneous fleet and each device runs the command its model declares; the edge dispatch and the command declaration live in templates / nodes.