Scaling and deployment

Omniglass is one Go binary, and that is a packaging decision, not a scale ceiling. The same artifact runs an all-in-one container on a laptop and a horizontally-scaled fleet on Kubernetes; you scale by topology, not by swapping products.

Partial

Built today: the single Go binary (hand-written commands: server, migrate, bootstrap, token, set-password, seed-dev, and node run), the embedded NATS server (in-process nats-server with JetStream, internal/cli/server.go), the node run mode, the multi-arch container image, and the Helm chart. Today one stream and one serial consumer do confinement and persistence inline; the design fences below mark the parts of this page that are target design, each naming its tracker. See implementation status.

Two run modes, one binary

The binary is a modular monolith: one codebase, one artifact, modules behind clean seams (the Storage Gateway is the only path to the database, coordination rides NATS, collection runs at the edge). It runs two ways, no fork:

• All-in-one (the modular monolith). One process runs every role, with Postgres and NATS embedded (below), against nothing external. The small-fleet case: download, run, done.
• Split by run mode (Kubernetes). The same binary launched per mode as separate Deployments, against an external Postgres and an external NATS cluster. A Helm chart wires it up, and each role scales independently.

Splitting a mode onto its own pods is a deployment choice, not a rewrite: the modules already talk over NATS and the gateway. The roles:

• server: the public HTTP API (API) and the views read path; it serves the SPA embedded in the binary (go:embed), so the web UI is not a separate service. Stateless.

Target design, tracked in #57

• worker: the JetStream consumers (rule engine, reconcile, notify, workers). Stateless competing consumers.
• controller: the leader-elected singletons (the clock and the CDC publisher, below). A role, not necessarily its own pod.

• node: collection; runs at the sites (outside the cluster) and connects back, plus a central node for cloud-API and SaaS sources (placement: central, nodes).

Embedded services (single-binary mode)

In all-in-one mode the binary brings its dependencies up in-process, so an operator runs one container, zero external setup:

• NATS + JetStream, embedded as a library (nats-server in-process, file-backed). The app is always a NATS client; embedded versus external is a config flag, not a code path.

Embedded Postgres, tracked in #19

• PostgreSQL, embedded as a managed subprocess (embedded-postgres): a real Postgres, so logical decoding (the CDC bridge below), JSONB, partitioning, and the exclusive-arc CHECK constraints behave identically to at-scale. Pinned to Postgres 18.3.0 or newer for ARM and x86. Not SQLite, which has no logical replication and would fork the data layer into a second, lesser architecture.

Coordination: NATS moves, Postgres remembers

Postgres is the relational system of record (entities, samples, events and alarms, audit, and the queries the cascade, fusion, views, and scope need). NATS (JetStream) is the nervous system: work distribution, the durable command queue, the telemetry buffer, and fan-out.

The JetStream KV and object store, tracked in #529

JetStream KV (config, locks, leader-election) and an object store for internal artifacts ride the same substrate; user files stay on the content-addressed blob store (built: internal/blob, Postgres-backed), not the object store.

Target design, tracked in #430

The two meet through change data capture: a single leader-elected CDC publisher bridges committed Postgres changes onto JetStream; Postgres is never a message bus, it only emits its changes. The mechanism is described once, on messaging.

Target design, tracked in #430

Inter-service communication

Service-to-service traffic rides two lanes on the one JetStream bus: the NATS-native data lane for samples, and the Postgres-first, CDC-out record and state lane for events, alarms, actions, and operator mutations. The lane model is described once, on messaging; here is how the pieces deploy:

Target design, tracked in #57

Horizontal scale: what replicates

• server is stateless: replicate it behind a load balancer; state lives in Postgres.
• workers are JetStream consumers: a work-queue stream delivers each message to exactly one consumer, so adding replicas adds throughput (NATS is the coordinator, workers).
• edge nodes: one or many per site, connecting back; adding sites adds nodes (nodes).
• singletons (the clock and the CDC publisher): leader-elected via a NATS KV lock, exactly one active, failover by re-election. No separate election service.
• the control-plane consumers (worklist, heartbeat) are today singletons by construction: one server holds each per-verb wildcard subscription (ADR-0081). The HA fork, queue groups versus worklist reassignment, is named and deferred to the day a second server exists.

Platform configuration

Configuration is two tiers, and platform settings are deliberately centralized:

• Bootstrap (env, optional). The irreducible minimum needed before the database exists: the Postgres DSN, the NATS embed-or-external choice and address, the SecretProvider key, the run mode, and the listen address. In all-in-one mode these have working defaults, so a desktop run needs no configuration at all; env vars override.
• The platform settings store (one place). Everything else lives in the single, audited settings engine: feature flags, buffer and retention defaults, CDC routing, integration settings, UI defaults, official-registry overrides, resolved most-specific-wins down the principal hierarchy. An operator settings file (settings.json or YAML) is the GitOps layer, read into memory at boot and mounting cleanly as a Kubernetes ConfigMap (and a future operator); only the override an operator sets through the API is persisted in Postgres (audited), while the file and the embedded defaults are recomputed each boot, so restore is a delete and the file never drifts into a second authoritative copy (ADR-0033).

This is distinct from fleet config and variables, which describe the fleet and resolve down the cascade; the settings store describes the platform itself.

Partial: the settings engine’s platform level

The settings store is built at the platform level (the install-wide rung, renamed from global by ADR-0057): the pure resolver, the setting_override table, the admin API, the two settings:<action> permissions, and the Admin settings page, with ui.theme wired through to re-theme the SPA (see settings). The group and user cascade rungs, and the platform-domain settings (retention, CDC routing, integrations) named above, are the fast-follow.

Target design, tracked in #57

Vertical scale and high availability

Replicas are the HA story: the server and worker tiers have no single point of failure, the singletons re-elect on the NATS KV lock, Postgres HA is the database’s concern (CNPG, a managed cluster), NATS HA is the JetStream cluster’s, and the edge survives a WAN outage on its own (the bounded buffer plus the durable command queue, nodes). Vertical scale is the simple first lever; horizontal removes the ceiling.

Multi-tenancy: per database, per account, per deployment

Tenant isolation is physical, not a row predicate: a tenant is one database, one NATS account, and one deployment. There is no tenant_id column anywhere, no shared row store, and no shared subjects, so per-database isolation (storage) and per-account isolation (messaging) are the same boundary. The data model stays single-tenant-shaped; multi-tenancy lives at the orchestration layer (CNPG-per-tenant). One noisy or compromised tenant cannot reach another: there is nothing shared to reach across (identity and access).