Files and blobs

Partial

The first slice (ADR-0029, epic #242, #244) is built: the content-addressed blob store as a Storage Gateway primitive (default pgblobs, dedup on the hash), the file handle CRUD over the API, generated CLI, and typed client, and the Files operator directory, gated by file:<action> plus the per-file sensitive flag. Deferred: attaching a file to entities and types (#242), async mark-sweep GC, the S3 and disk backends, tags-on-files (#191), and richer search; the resource-classification + clearance lattice subsuming the binary sensitive flag is its own epic (#243).

Files keep the opaque bytes that go with a fleet (a firmware image, a config dump, a runbook, a packet capture) searchable and deduplicated: a file handle over a content-addressed blob store, behind the same Storage Gateway as everything else.

Two layers: the file handle and the blob

• file is indexable metadata: a filename, content-type, size, sha256 (no tags column yet, deferred to #191), the searchable handle an operator references; it owns no bytes, pointing at a blob by hash. The filename is a human label with an extension (codec-firmware-2.1.4.txt), deliberately not a triad name: it takes no name rule, carries no uniqueness, and the handle is addressed by its uuid, which is why file is declared “human identifier, not a name” in the identity shapes.
• the blob store holds the bytes, content-addressed by sha256: the hash is the key, so identical bytes are one blob.

Splitting them means search and inventory never touch bytes, and one blob can back many handles.

Tags on files, tracked in #191

file tags will reuse the tag key registry (so category means the same thing on a firmware image as on a component, config and credentials) but bind as a flat per-file set: a file is not on the structural exclusive-arc, so there is no parent to cascade from. The vocabulary is shared; the cascade is not.

Access: a permission and a sensitive tier, no placement

A file carries no fleet placement: unlike a secret, which is for one component or system and sits on the exclusive arc, a file relates to many things (a firmware image documents many device types) or to nothing, so its locality is its future attachments (a many-to-many relation), not an owner column. A file is tenant-wide, gated by two existing layers:

• the file:<action> permission on every route (file:create, file:read, file:delete; download rides file:read);
• a per-file sensitive flag reusing the secret sensitivity tier (ADR-0025): a flagged file is lifted to the :admin tier (file:read:admin), hidden from a lister and a non-disclosing 404 to a reader without it (a competitive quote is a sensitive global file, a firmware image an ordinary shared one).

Unlike a secret, the flag defaults false, and file is not in the sensitive-resource set, so the viewer floor (*:read) reads ordinary files; only the flag fences the confidential ones.

The resource-classification and clearance lattice, tracked in #243

When the resource-classification + clearance lattice lands, it subsumes this binary flag (a 2-rung case of it) across files and secrets alike; the external-principal class it introduces is what will keep an outside integrator from ever seeing an internal file.

Content-addressing earns four properties

A blob is keyed by the hash of its bytes, not a UUID, which buys:

• dedup: identical bytes collapse to one blob (two operators uploading the same firmware);
• integrity: the hash verifies the bytes on read, tamper-evident by construction;
• immutability: bytes cannot change without changing the key;
• backtest-stability: an event referencing a hash still resolves under a backtest.

Hash-refs from the signal lanes (the log_line body, the collection.failed raw), tracked in #242

So rows reference a hash, never inline bytes (inline bytea would kill hash-ref stability and bloat the firehose row). Small structured values stay inline in the row’s jsonb; large or opaque payloads become a blob hash-ref (a dedicated indexed blob_sha256 column, so GC can probe it): a big log_line body, and especially a collection.failed event’s raw when the wire payload is large (a full SNMP walk, a big HTTP body, a capture). Raw stays inline when small; the size threshold is the switch.

Open question

The inline-versus-blob size threshold: one global cutoff, or per-kind (raw versus log body versus operator upload).

Dedup is database-scoped

The blob key is sha256, the bare content hash, no tenant_id: isolation is per-database (a database per tenant), so dedup is global within a database and one tenant can never detect another’s content by hash collision, because the blobs never share a store; the efficiency cost is the right price for physical isolation.

Backends, swappable behind the gateway

The bytes live behind the Storage Gateway, so the backend swaps with no model change (the same seam as the columnar and object tiers):

• default: pgblobs (a dedicated Postgres blob table), the single-binary, no-external-dependency story;

Alternate blob backends, tracked in #248

• scale: an S3-compatible object store;
• disk for local and dev.

The file and the hash reference are identical across backends; only storage_ref resolution differs.

Open question

Chunking and streaming for very large blobs (firmware images, captures) on the pgblobs backend.

Reference-counted GC, not age-based

Async mark-sweep GC, attachments, and the reference lanes, tracked in #242

A blob is collectable only when no live reference points at its hash AND a grace or retention floor has passed (age-based GC alone is wrong: dedup means a blob uploaded long ago can be the one a recent event references). References come from a file handle, a large log_line body, a collection.failed raw hash-ref, and an attach event (an event or audit_log row recording “this component was attached to this file at T”). They disappear when a file is deleted or a referencing event ages out (a retention partition drop), so GC is coupled to retention: dropping a partition releases its references.

Mechanism: index-probe mark-sweep by default. GC enumerates blobs past the grace floor and probes the indexed hash-ref columns on the referencing tables. A maintained refcount column or blob_ref table is a measured optimization, earned only if the per-blob probes profile too expensive; the grace floor is the safety margin so GC never races a just-written event.

Open question

The grace-floor duration relative to the backtest window (long enough that a prospective backtest re-deriving over the window cannot reference a collected blob).

Storage

The handle and the content-addressed bytes; the physical layout (the gateway, GC) is above and on storage.

file (Partial) is the searchable metadata handle, pointing at a blob by hash: tenant-wide, sensitive per the access section above.

file (content): generated from the migrated schema by make gen

Column	Type	Constraints	Notes
id	uuid	PK, default uuidv7()
name	text	not null
content_type	text	not null
size	bigint	not null
sha256	text	FK → blob.sha256, not null	The content address of the bytes this handle names
sensitive	boolean	not null, default false	Lifts the row to the :admin tier
created_at	timestamp with time zone	not null, default now()
updated_at	timestamp with time zone	not null, default now()

blob (Partial) is the content-addressed bytes: dedup on the hash, default pgblobs (inline bytea), S3 / disk behind the same blob.Store seam. Content type lives on the file, because content-addressing is about the bytes. A file delete frees its unreferenced blob synchronously (dedup-aware); async mark-sweep GC is deferred.

blob (content): generated from the migrated schema by make gen

Column	Type	Constraints	Notes
sha256	text	PK	The content address; dedup key across every referencing file
bytes	bytea	not null
size	bigint	not null
created_at	timestamp with time zone	not null, default now()