Expressions
PartialExpressions let an operator reshape and judge collected values in plain text wherever the
platform needs a small computation, with exactly one language to learn: every site goes through
one engine, Omniglass expressions, built on Expr
(expr-lang/expr, pinned at v1.17.8). The engine is
internal/expression (#524); its first consumer is the event rule’s criteria
(alarms and actions).
Compiling, once
Section titled “Compiling, once”A site declares its environment as a Go struct (the names an expression may use, their types and
docs), the type an expression must yield (a criteria yields true or false), the builtins it admits
and its run budget. An expression compiles against that declaration: a name outside it, a result of
another type, or a parse error is refused with a diagnostic carrying the line, the column and an
operator-readable message (a nested binding is named by its path, component, never by a Go type). A
name nothing binds is said by name (unbound, so an editor can offer to bind it), and a let that
would shadow a binding is refused where it is declared. Expr trusts a function call’s type on its final
check without looking inside its arguments again, so the engine clears it, and a fault inside a call’s
argument (an unknown name in max(fan)) is never passed over.
A compiled program is cached by its source and its site’s declaration in a bounded cache, so a
hot path (a criteria judged per sample) never recompiles; an event rule also holds its compiled
criteria per rule version, a failed compile included.
Safety
Section titled “Safety”Expressions are sandboxed: no I/O, no network, no unbounded loops, bounded execution.
- An allowlist of builtins. Every Expr builtin is off, and each site declares the builtins it
admits. A builtin is admitted only once vetted: its cost linear in what the run is charged for,
its output no larger than its inputs. The criteria set is comparison, arithmetic,
in,contains/startsWith/endsWith,matcheswith a constant pattern (capped in compiled size, and charged its subject’s length times that size),len,abs/ceil/floor/round/min/maxon numbers only,lower/upper/trim/trimPrefix/trimSuffix, andall/any/none/one/count/filterover a bound list with no predicate nested in another. Equality andintake scalars and lists of scalars (and a key in a map), never a record, a map, or a list of lists, whose deep comparison no charge covers. - List functions (#1018) are vetted the same way, each with its cost argument (one charged pass
over a list, one value out), and admitted by name where a site binds lists:
maxandminover one list of numbers (or over numbers),avg,sum(with or without a predicate),majority(the most common text or bool, its strings charged their bytes, a tie to the value that sorts first), and the one-listanyandallover bools beside their predicate forms. A list function given anything but its list is refused at compile time on the argument. Their names, like every builtin’s, can never be bound, so admitting one later never changes what a stored expression means. Expr parsesany,allandsumas predicate builtins only, so a call of one with a single argument is told apart before parsing, lexically, and renamed in place to the engine’s own function of the same length, which keeps every position in the source where the author wrote it. Builders and encoders (map,reduce,repeat,replace,join,uniq,flatten,toJSON, the range..) are never admitted: their cost is native work that no input charge bounds. - Determinism. The clock and zone builtins (
now,date,duration,timezone) can never be vetted; a time an expression needs is bound by its caller as data. An expression calls builtins only, never a method or a bound value, and a site’s environment holds concrete data types (no interface, so no value whose behavior is chosen at run time). A$name (Expr’s dynamic$env, or the engine’s own plumbing) is refused, so a site’s bindings are the whole namespace. - Caps and budgets. A source is at most 4096 bytes and 1000 syntax nodes, and a list or map literal at most 64 entries. A run is bounded by its site’s budget of what it allocates and what it walks, deterministically; a criteria’s is small, and no site’s exceeds a fixed ceiling. A wall-clock deadline backs the budget for charged work only: it cannot interrupt a native call, which is why the allowlist exists.
- Values are data. A value bound in the environment (a sample’s text, however hostile) is evaluated literally and never executed. Integer arithmetic wraps on overflow, as Go’s does, rather than failing.
- Compiled unoptimized, and refused rather than crashed. Expr’s optimizer is off: in v1.17.8 it
rewrites
filter(list, p)[0]and[-1]intofindandfindLastand loses the record’s field table on the way, so a field read on the result (filter(members, .has.a)[0].value.a) panicked in Expr’s compiler. It comes back on when upstream is fixed (#1006). Nothing is charged differently without it: every charge wraps its operand before Expr would fold it. A panic in any compile stage, the engine’s own or Expr’s, is refused as a type error naming the stage and the construct it was on, never a crash or a nil pointer.
Sites: the bindings, builtins and budget of each
Section titled “Sites: the bindings, builtins and budget of each”Each place an expression is written is a site, declared once in code. The tables below are
generated from those declarations (make gen), and POST /expressions:validate (omniglass expression validate, gated rule:read) answers from the same ones: given a site, a source and,
for criteria, the sample key (for calc, the output key and the rule’s inputs), it says whether the
expression compiles, with each diagnostic’s line and column, and returns the site’s bindings,
builtins and budget. The calc site (#995) is the first whose bindings are authored with each
expression: a calc rule’s inputs, each bound by its alias as the list of its current values (#1018),
from a struct type built when the rule is written, so its table describes them rather than listing them,
with the list functions it admits and the names an alias may not take (both generated, and the
console’s alias check is generated from the same list). The console’s expression field
(the rule drawer’s Write as an expression, #960, and the calc rule’s reduce) is that route’s face, a
code editor on CodeMirror 6 (#1000) driven entirely by the route’s answer: Expr highlighted as it is
written, completion of exactly the site’s bindings and builtins (as a word is begun, or on Ctrl-Space),
a binding’s doc on hover, and each diagnostic underlined where it sits as well as pointed at beneath under
its line and column, with the site’s bindings and functions listed under the field. The field asks for
the site’s vocabulary while the source is still empty, so the first word completes too. A site’s form
may offer start-from templates (the calc rule’s list functions, inserted with the cursor in the slot)
and an action on a name nothing binds (the calc rule’s Bind it to a key, in the fault’s tooltip and
beside it). The editor is named by its field’s visible label. The editor is a chunk of its own, loaded
only by a page with an expression site; a page that cannot load it says so and refuses the source. Built today:
calc
: A calc rule's reduce: one value of the output key's type, from the rule's aliased inputs on one owner. The output key types the result, so a key is required.
Bindings: Each of the rule's inputs by its alias, as the list of its current values, typed by its key (a metric is a number, a property text, a bool property a bool). For a component owner, one value per instance: the named instances, the aggregate instance only when it is the key's one, an instance not heard from in an hour left out. For a system owner, every role-bound member's values, ordered by member name then id; and members: one record per member, holding id, name, value.<alias> (the member's own current value, the type's zero when it has none) and has.<alias> (whether it holds one), so one member's inputs stay together. A list function (avg, max, min, sum, majority, any, all) folds a list into one value. The result is number for a metric output, text for a string or json property output or bool for a bool property output.
| List function | Takes, and why it is bounded |
|---|---|
all | one list of bools: one pass over the charged list, output a bool (the predicate form is the builtin's) |
any | one list of bools: one pass over the charged list, output a bool (the predicate form is the builtin's) |
avg | one list of numbers: one pass over the charged list, output a number |
majority | one list of text or bools: one pass over the charged list, its strings charged their bytes, counting into a map no larger than the list, output one of its values |
max | numbers, or one list of numbers: one pass over the charged list, output a number |
min | numbers, or one list of numbers: one pass over the charged list, output a number |
sum | one list of numbers, or a list and a predicate yielding one: one pass over the charged list, output a number; a predicate may not nest another |
The 92 names an alias may not take (keywords, builtins, list functions and the site's own bindings)
abs, all, and, any, avg, bitand, bitnand, bitnot, bitor, bitshl, bitshr, bitushr, bitxor, ceil, component, concat, contains, count, date, duration, else, endsWith, false, filter, find, findIndex, findLast, findLastIndex, first, flatten, float, floor, fromBase64, fromJSON, fromPairs, get, groupBy, has, hasPrefix, hasSuffix, if, in, indexOf, int, join, keys, last, lastIndexOf, len, let, lower, majority, map, matches, max, mean, median, members, min, nil, none, not, now, one, or, reduce, repeat, replace, reverse, round, sort, sortBy, split, splitAfter, startsWith, string, sum, system, take, timezone, toBase64, toJSON, toPairs, trim, trimPrefix, trimSuffix, true, type, uniq, upper, value, values
Builtins it admits: abs, all, any, avg, ceil, count, filter, floor, len, lower, majority, max, min, none, one, round, sum, trim, trimPrefix, trimSuffix, upper
. A run may walk 100,000, allocate 50,000 and is stopped after 25ms of charged work.
criteria
: An event rule's fire_criteria and clear_criteria: true or false for one arriving sample. The sample key's lane and data type type the value, so a key is required.
| Binding | Type | What it holds |
|---|---|---|
value | float64 for a metric; string for a string or json property; bool for a bool property | The sample's value, typed by the key: a metric's number, a property's text, or a bool property's truth value |
key | string | The sample key the rule reads: a property or a metric, by name |
instance | string | The sample's instance (an endpoint), empty when it has none |
component.name | string | The component's name |
component.type | string | The component's type, by name; empty when its product names none |
Builtins it admits: abs, all, any, ceil, count, filter, floor, len, lower, max, min, none, one, round, trim, trimPrefix, trimSuffix, upper
. A run may walk 10,000, allocate 10,000 and is stopped after 5ms of charged work.
Diagnostics: clear messages and quick fixes
Section titled “Diagnostics: clear messages and quick fixes”Expr’s own refusals are accurate but speak the checker’s language (sum(members, .has.t) > 0 is
refused as an operator mismatch at the >, when the mistake is the sum), and an expression can compile
and still probably be wrong. So the engine keeps a registry of diagnostics (#1021,
internal/expression/diagnostics.go): each common mistake is one declared rule with
- a stable code (
sum-of-bools), a severity (an error refuses the expression; a warning, probably wrong, and a hint, could be simpler, ride along with one that compiles and never refuse it), and the sites it applies at (criteria, calc, or every site; a site names itself on its environment); - a trigger: a refine rule runs when the engine refuses an expression and recognises the shape behind the refusal, replacing the message with a clearer one at the place the mistake is (it explains a refusal, so it is always an error); a lint rule runs on an expression that compiles, on its checked tree, and flags a likely mistake;
- a matcher, a pure function over the typed tree (and, for a refine rule, the refusal), returning the positions it found;
- a message, and an optional fix: one text edit (a range and its replacement) an editor offers as a quick fix;
- fixtures beside it in Go: expressions it must give its diagnostic for and must not, each compiled against a site-shaped environment, with the fixed source a fix must yield (which must compile without it). A rule without both kinds of fixture refuses to register, and a fixture that disagrees with its rule fails the build.
Compile and Check run a site’s rules once per compile, through expr.Compile itself (a refine rule
reads the refused expression’s tree, typed as far as Expr’s checker got, through a visitor that only
records its root), so a cached program carries its warnings and hints, and a matcher that panics is a bug in the rule: the expression is judged as if the rule were absent,
never refused by it and never crashing the server.
POST /expressions:validate answers each diagnostic’s severity, and a registered one’s code and fix,
so the console’s expression field shows them where they arise (#1053): an error is underlined with a
wavy red line and refuses the source, a warning with a wavy amber one and a hint with a dotted line, neither
marking the field invalid nor stopping a save. Beneath the field each diagnostic names its severity, links
its code to its entry in the table below, and offers its fix as a button (a keyboard user reaches it with
Tab; in the editor, F8 moves to the next diagnostic and Mod-Shift-M lists them with their fixes). A fix is
an edit of the author’s own, so Ctrl-Z takes it back. A stored rule’s warnings show on its blade as a note,
never an alert.
The first rules (#1054) are the mistakes authors made first: counting with sum, comparing a measured
number to a decimal with ==, spelling all as a count against a length, and an input the reduce never
names (a calc site declares its aliases as the expression’s inputs). Each is in the table below, with its
severity and fix.
Every registered diagnostic, generated from the registry (make gen): its code, its severity and when it
runs, its sites, a summary (the diagnostic itself says it in the expression’s own terms), and its examples,
read from its fixtures with the site, inputs and result each is written against: what it flags, with what
its fix gives, and what it leaves alone, including another refusal the engine makes on its own. The build
holds the rule to each. Built today:
| Code | Severity | Sites | In short | Examples |
|---|---|---|---|---|
sum-of-bools | error explains a refusal | every site | sum adds numbers; to count matches use count(...) |
|
float-equality | warning flags an expression that compiles | every site | a measured number seldom equals a decimal exactly; compare a range instead, such as abs(value - 1.5) < 0.05 |
|
count-equals-len | hint flags an expression that compiles | every site | count(xs, p) == len(xs) is all(xs, p) |
|
unused-alias | warning flags an expression that compiles | calc | an input the expression never names is read for nothing; use it, or remove the input |
|