Store

Summary

The store contains immutable payloads, user-facing object metadata, the BuildKey → ObjectHash and ReuseKey → ObjectHash reuse mappings, and mutable refs from a name to its object. Run logs and scratch directories are configured separately and conventionally placed under the store. It is a content-addressed store (CAS): payload identity is derived from normalized content, not from the path or name used to reach it.

The store root is the directory named by the request’s store field — an absolute path to an existing directory, used as-is (bobr adds no implicit .bobr/ layer).

For the mental model behind objects, keys, and CAS, see Concepts. This document is the precise reference for the store’s identity model, reuse rules, and on-disk layout.

Identity Model

Concepts introduces the three identities (object_hash, build_key, reuse_key); this section is their normative definition.

Every recipe has a build_key.

For Source, build_key is the declared object_hash reinterpreted as a build key. Source recipes have no inputs.

For a builder recipe, build_key is computed from:

• builder tag
• builder implementation-version token
• canonical config digest, after typed parsing and defaults
• direct dependency build_keys keyed by input name

The version token includes the common build-semantics version and the builder’s implementation version; architecture-dependent builders also include the build architecture. It is not simply the Bobr release version.

The fingerprint includes both each input name and its hash, in lexical input-name order, independent of JSON field order. This is distinct from the builder’s input consumption order. The key is SHA-256 of the canonical invocation fingerprint.

build_key identifies a recipe; the store maps it to the object the recipe produced (builds/<build_key>). For a builder recipe, a hit on that mapping can be used before looking through the recipe’s direct inputs.

Once the identities of a builder’s direct input objects are known, its reuse_key is computed from:

• builder tag
• the same implementation-version token
• the same canonical config digest
• direct dependency object_hash values keyed by input name

The same lexical input-name order is used. Build and reuse fingerprints use different invocation schemas to separate the two key domains. Determining an input object’s identity does not require its content to be present locally.

reuse_key is independent from the particular dependency invocations that produced those input objects. Different build graph fragments can therefore reuse a stored result when their direct input payload identities match.

Executing a builder or materializing a source produces one payload object. The payload is addressed by object_hash.

The same object_hash also keys a user-facing object metadata record for that payload. Different builder recipes can share one record when they intentionally stage the same payload. The record is not part of object identity or cache resolution.

Recipe names do not participate in object identity, build_key, or reuse_key; input slot names do.

Reuse Model

For a builder recipe, the working-store lookup starts with an exact build_key hit in builds/. A complete exact hit skips everything below it. On an exact miss, bobr resolves possible object_hash identities of the direct inputs and computes the corresponding reuse_key values. It then checks those keys in reuses/. This identity resolution may happen before input content has been copied into the working store. A mapping supplies a candidate hash, not a guarantee of available content. If no candidate can be reused, bobr realizes complete direct inputs, checks reuse again with their actual hashes, and executes the builder if no reusable result is available.

If a reuse hit provides an object but the current build_key mapping is missing, bobr publishes that mapping and reuses the object. A builder can also publish every reuse key in the current candidate set that resolves to the chosen object.

Local builds/ and reuses/ each hold one current object per key. A key does not prove that a builder is deterministic: publishing a different result for the same key replaces that mapping, while the previous payload remains in the store.

For Source, there is a build_key but no reuse_key.

Source realization asks whether its declared object_hash is complete in the working store, then whether configured content sources can provide it. If not, the Source origin is materialized. Object records are not read during this lookup. On a source hit or successful materialization, bobr writes user-facing metadata and creates or repairs the source’s builds/<object_hash> mapping.

If source materialization produces a different object than the declared object_hash, the actual object is still imported into objects/, but no object metadata or source builds/<object_hash> mapping is written, and the source import fails with the actual hash.

Complete content

An ordinary object is complete when its payload is present as a file or directory in objects/. An fs-tree additionally requires every fs-file referenced by its canonical manifest. Neither an object record nor an already materialized fs-trees/ cache entry is required for completeness.

Working-store completeness checks inspect presence and file types and parse manifests; they do not rehash all payloads on every cache hit. Store content must remain immutable. Secondary transfers verify the expected object and fs-file identities before publishing a complete result.

Secondary provider capabilities

A request can additionally name local or remote backends (see Request). It exposes each backend through one or both independent capabilities:

• a mapping provider answers BuildKey and ReuseKey queries with candidate ObjectHash values; it supplies identity, not object bytes;
• a content provider supplies an object’s bytes by ObjectHash. Local providers can hardlink objects and every referenced fs-file, or copy both ordinary objects and complete fs-tree closures into independent working-store inodes. A hardlink import shares regular-file inodes while recreating directory containers and symlinks; a copy import shares no regular inode, even when both stores are on one filesystem. Both transports verify the expected object or fs-file identity and preserve the logical metadata that participates in it before atomic publication. Hardlink repositories require the repository and working-store objects/ directories to share a filesystem, as must their fs-files/ directories; bobr validates the two pairs independently before scheduling realization and never falls back to copy. Remote providers download and verify ordinary objects or complete fs-tree closures, fetching missing fs-files before publishing the imported result in the working store.

Configuration enables the mapping and content-source capabilities explicitly; neither implies the other. Profile syntax may combine both capabilities in one entry, but request lowering separates them again.

The Realizer consults working-store mappings before repository mappings, and checks working-store content before secondary content. Providers are consulted in their declared order within each capability. A hash learned from a mapping provider can be supplied by a different content provider. Mapping lookup never opens object records.

The remote protocol, signed metadata, wire representations, and publication workflow are specified in Remote repositories.

An imported result is complete working-store content, not a borrowed path into the repository. With copy transport this follows from independent inodes. With hardlink transport the repository and working store have independent directory entries naming shared immutable inodes, so unlinking the repository names does not remove the working names. A subsequent offline request may omit the repository entirely. Local store layouts and provider settings are validated at startup even when every requested result is already local; remote metadata is fetched lazily when a provider is consulted.

Store Layout

The filesystem layout mirrors the identity model:

<store>/
  objects/
    <object_hash>
  reuses/
    <reuse_key> -> ../objects/<object_hash>
  builds/
    <build_key> -> ../objects/<object_hash>
  object-records/
    <object_hash>.json
  object-refs/
    <name> -> ../objects/<object_hash>
  fs-files/
    <first-two-hex-digits>/
      <fs-file-hash>
  fs-trees/
    <manifest-object-hash>/
  fs-tree-refs/
    <name> -> ../fs-trees/<manifest-object-hash>

A run’s log and work directories are not part of this layout: the request names them, and the caller creates them (see Request). By convention they live under the store — <store>/logs/<run-id> and <store>/work/<run-id>, which is what bobr-build.sh does by default — and the work directory has to be on the store’s filesystem, since build output is published out of it by renaming and hardlinking. Their contents are:

<logs>/
  context.json
  recipe-catalog.json
  events.jsonl
  index.jsonl
  <00000000>-<tag>[-<name>]/
    meta.json
    events.jsonl
    raw/
<work>/
  <00000000>-<tag>[-<name>]/

• objects/ holds payloads addressed by object_hash.
• object-records/ holds user-facing metadata records addressed by object_hash.
• reuses/ maps a reuse_key to its object (builder recipes only).
• builds/ maps a build_key to its object.
• object-refs/ holds human-facing refs from recipe name to the latest successful object for that name.
• fs-files/ holds regular-file payloads referenced by fs-tree manifest objects. Each file already carries its logical ownership and mode in the build namespace; materialization preserves them through hardlinks. Its hash covers both metadata and bytes; see Filesystem trees.
• fs-trees/ caches materialized filesystem roots for fs-tree manifest objects.
• fs-tree-refs/ holds human-facing refs from recipe name to the latest materialized filesystem root for that name.

objects/<object_hash> is the payload itself, either a file or a directory. Concrete directory payload formats are builder-specific. For example, the OCI registry source handler realizes imported images as OCI image layout directories.

Filesystem tree builder results store a canonical fs-tree manifest text file as the object payload. Regular file entries in that manifest reference payloads stored under fs-files/. Materialized roots under fs-trees/ are cache entries created on demand for builders that need a filesystem root path; a fresh build output can also populate this cache while its files are interned. When a named builder input asks for a filesystem root, the runtime also updates fs-tree-refs/<name> -> ../fs-trees/<manifest-object-hash>. These refs are for inspection only. Runtime lookup uses object identity and the fs-trees/ cache directly; it does not read fs-tree-refs/. If the same name is later materialized from a different manifest object, the old current ref is rotated into an mtime-suffixed generation ref before the new current ref is installed.

Remote metadata has a separate auxiliary cache, defaulting to <store>/repository-cache and configurable as secondaries.repository_cache. This is not CAS payload storage: successfully imported objects and fs-files live in the normal directories above. See Request for configuration and Repository cache for its shared client/utility use.

Generic CAS objects may contain non-UTF-8 filesystem names. Such objects can still be imported and addressed by object_hash. Fs-tree objects are UTF-8-only because their manifest paths and symlink targets are JSON strings.

object-records/<object_hash>.json is user-facing metadata for people and store-inspection tools. Its current schema is bobr-object-record-v4; it contains:

• object_hash — the object the record describes
• build_key and inputs — producer information supplied by the first successful writer of this record
• run_id — optional; the store run that recorded it

The record is idempotent: an existing record is retained rather than rewritten. Consequently its producer fields can be neutral metadata written while recording an already-present or imported object; they are not authoritative provenance for every mapping that later reaches this object.

builds/<build_key> and reuses/<reuse_key> are canonical symlinks whose targets encode an object_hash as ../objects/<object_hash>. Mapping lookup validates and reads the hash from the symlink text without following its target or inspecting object content; content is checked separately when realization needs it. Publication creates these mappings only after the target object is complete in the store. Object records remain independent inspection metadata, just as object-refs/ and fs-tree-refs/ are human-facing views rather than lookup inputs.

<logs>/<serial>-<tag>[-<name>]/raw/ stores raw per-subject log files such as captured tool output. <work>/<serial>-<tag>[-<name>]/ is the matching per-subject scratch directory. Scratch directories are removed after execution on a best-effort basis; cleanup failures are logged as warnings and the scratch directory is left in place.

Object Refs

Every recipe carries a name. The name enters no identity computation — not object_hash, build_key, or reuse_key; it is used for the refs described here and for diagnostics and logs.

When a named recipe successfully resolves to an object, the current object ref is updated:

• object-refs/<name> -> ../objects/<object_hash>

This object-refs/ rule is the same for every object kind. Filesystem tree builder results store the manifest itself as the object payload. The object ref never points directly at fs-files/ or at a materialized fs-trees/ cache directory. When an object has been successfully published, its user-facing metadata is normally available as object-records/<object_hash>.json; object-ref lookup does not depend on that metadata.

Both object-refs/ and fs-tree-refs/ are inspection aids, not cache lookup inputs. The latter are created when a filesystem root is materialized for a named input.

If the current object ref already points at a different object, the old current ref is rotated into a timestamp-suffixed history ref.

Local store lifecycle

Updating a mapping or name ref does not delete the previous object or its fs-files. Bobr currently has no local store garbage collector: old payloads, records, and history refs remain until explicitly removed. Immutable shared files must not be modified in place. Remote repository retention and garbage collection are separate mechanisms described in Repository CLI.

Logging

Each run writes:

• one run-level structured event log under <logs>/events.jsonl
• one workspace index under <logs>/index.jsonl
• per-subject logs under <logs>/<00000000>-<tag>[-<name>]/

When the standard bobr-recipes/bin/bobr-build.sh driver starts the run, it also writes two inspection sidecars at the run-log root:

• context.json records the ordered goal recipe names, run outcome, bobr --build-info, and the recipes checkout commit and dirty state when Git provenance is available;
• recipe-catalog.json contains only the reachable recipe names and builder tags needed by store-inspection tools. It deliberately omits builder configs, origins, provider configuration, and credentials.

The driver creates no sidecars for a dry run. It records running before invoking Bobr and atomically replaces that state with success or failed after Bobr returns, so an interrupted process leaves an identifiable incomplete run. These files are diagnostic metadata, not inputs to cache lookup or object identity.

The run id comes from the request; bobr-build.sh derives it from the local <YYMMDDhhmmss> timestamp and appends .1, .2, and so on when a directory of that name is already taken. A subject with an allocated workspace receives a serial number for its log directory name. The serial is an internal allocation detail; the full original tag, recipe name, subject key, and workspace paths are stored in that subject’s meta.json. Recipe subjects use build keys; for Source, that is the declared object hash. Repository metadata and content transfers use their own activity identifiers, not necessarily build keys.

The run-level event log is the full audit log of the run. Subjects with dedicated logs also write their events to their own events.jsonl; events without a bound subject workspace, including repository transfers, appear only in the run-level log. Raw logs created by builders are written under the subject’s raw/ directory. The event record format, the closed status vocabulary, and the format guarantees are specified in Build logging.