`bobr-repo` command-line utility

This document specifies the design of the administrative bobr-repo command. The command publishes a complete local Bobr store to an S3-compatible remote repository, signs and commits repository masters, reports repository state, and performs garbage collection.

The remote wire format is specified separately in MASTER.md, OBJECT.md, FS_FILE.md, and TAR.md. The command uses the same library implementation as the Bobr remote-repository client.

Scope

bobr-repo has five subcommands:

bobr-repo init
bobr-repo prepare
bobr-repo publish
bobr-repo status
bobr-repo gc

init creates and configures a dedicated S3 bucket. The first signed master still creates the repository’s logical state. There is no fixed slot-count configuration and no general reconfiguration command. Slots are added and rotated explicitly by prepare.

The utility does not create signing keys. It reads an unencrypted Ed25519 PKCS#8 private key in PEM or DER form, generated by a general-purpose key-management tool such as OpenSSL. The key file must be a regular file with no group or other permission bits. Encrypted PKCS#8 and hardware-backed keys are outside the first implementation. The corresponding public keys use the standard SubjectPublicKeyInfo PEM or DER form and are supplied as trust anchors where a current master must be authenticated.

For example:

openssl genpkey -algorithm Ed25519 -out repository-key.pem
chmod 600 repository-key.pem
openssl pkey -in repository-key.pem -pubout -out repository-key.pub.pem

Deployment and credentials

A normal deployment uses two machines and two independently scoped S3 identities.

The build server runs prepare. Its S3 identity may read the repository and write immutable keys under:

b/
r/
lo/
lf/
o/
f/

It must not be able to replace /master. The build server does not possess the repository signing key.

A trusted workstation runs publish. It holds the Ed25519 private key and an S3 identity that may read the repository and conditionally create or replace only /master. It does not need permission to write or delete immutable data.

An administrative S3 identity used for gc may list and delete immutable keys, but does not need the signing key or permission to replace /master.

An S3 administrator runs init. This identity may create the dedicated bucket and replace its complete bucket policy and bucket-level Public Access Block configuration. It is more privileged than any steady-state publication identity and need not be present on the build server.

Ordinary Bobr clients use anonymous HTTPS and pinned public keys. They have no S3 credentials and no repository secrets.

The separation prevents compromise of the build server from disclosing the long-lived signing key or directly publishing a new repository state. It does not independently establish that software built by a compromised server has the intended semantics; doing that would require an independent rebuild or a separate attestation system.

Repository location

Commands which use the S3 API accept the repository storage prefix as an S3 URI:

--repository s3://bucket/optional-prefix

The mutable master is stored at master below this prefix. All immutable keys use the relative paths defined by MASTER.md.

init is the exception: because it owns the complete bucket policy, it accepts only a bucket root in the form s3://bucket. It rejects a repository prefix.

S3 endpoint, region, credentials, and session tokens come from the standard AWS credential and configuration chain. This permits AWS S3 and other S3-compatible services without a Bobr-specific credentials file. A non-AWS deployment can, for example, set AWS_ENDPOINT_URL_S3 and AWS_REGION in addition to its credentials.

Every command also accepts an optional repository-local CA bundle:

--ca-bundle CA-BUNDLE.pem

The file may contain one or more PEM certificates. They are added to the standard trusted roots rather than replacing them. The same additional roots are used for authenticated S3 requests and anonymous reads through the public master and data URLs. This allows one invocation to use a private-CA S3 endpoint and a publicly certified CDN, or to trust different private CAs listed in the same bundle. Hostname and IP Subject Alternative Name checks remain enabled.

The bundle is a local transport setting. It is not embedded in /master, is not part of repository identity, and is not stored in the metadata cache. Malformed, empty, or unreadable bundles fail before the first network request.

`init`

init prepares a dedicated bucket for a Bobr repository:

bobr-repo init \
    --repository s3://BUCKET \
    [--ca-bundle CA-BUNDLE.pem]

The command is idempotent. It creates the bucket when it is absent, then owns and converges these two bucket-level settings:

Public Access Block has BlockPublicAcls = true, IgnorePublicAcls = true, BlockPublicPolicy = false, and RestrictPublicBuckets = false;
the complete bucket policy grants anonymous s3:GetObject only for master, b/*, r/*, lo/*, lf/*, o/*, and f/*.

It grants neither anonymous bucket listing nor anonymous writes. Because the policy is replaced rather than merged, the bucket must be dedicated to one Bobr repository. init does not alter account-level Public Access Block.

Some S3-compatible implementations do not implement bucket-level Public Access Block. A recognized NotImplemented response is reported and does not prevent installation of the explicit bucket policy. Other failures are fatal. A missing bucket is distinguished from an inaccessible bucket: an access denial is never treated as permission to create a replacement.

If bucket creation succeeds but later configuration fails, the bucket is not deleted. A newly created S3 bucket is private at this point, and rerunning init repairs the incomplete configuration. The command does not create a master, a slot, repository content, or signing keys.

Public data location

The public immutable-data URL is a separate value:

--data-base-url https://data.example.org/bobr/

It is embedded in the signed master. It may refer to the bucket’s public HTTPS endpoint or to a CDN or custom domain. Its relative key layout must expose the same immutable objects as the S3 repository prefix.

On the first preparation --data-base-url is required. On subsequent preparations it is recovered from the verified current master. If explicitly provided again, it must match exactly.

Trust anchors and key identifiers

Commands which authenticate an existing master accept one or more pinned Ed25519 public keys. Public-key files are not secret and may be installed on both the build server and the trusted workstation.

The utility derives the COSE kid as the complete SHA-256 digest of the raw 32-byte Ed25519 public key. The publisher derives it from the signing key and a reader derives it from each configured public key. Users therefore do not configure a second independent identifier.

The master remains the trust root. A public key included in repository data is never trusted merely because the repository supplied it.

The local store as publication input

prepare publishes the content and mappings already present in the specified local Bobr store. It does not build recipes or fill in missing parts of their dependency graph.

The publisher includes:

every canonical entry under objects/;
every canonical entry under fs-files/;
every valid build mapping under builds/;
every valid reuse mapping under reuses/.

It excludes materialized fs-trees/, logs, object references, filesystem-tree references, object records, temporary files, and other derived or user-convenience state. An fs-tree manifest is included because it is an ordinary object under objects/; its separately materialized tree is not.

bobr-repo does not accept recipe goals or traverse the recipe DAG to select publication content. Use a fresh store to limit publication to the intended build. An ordinary world request does not make every reachable recipe a goal: an exact hit on a high-level target can skip its dependencies. To rebuild the full graph, use a fresh store with content-only providers or no secondaries; this is a build configuration choice, not automatic behavior of prepare.

Before upload, the publisher validates the store layout, every mapping target, every ordinary-object identity, every fs-tree manifest, and the complete set of referenced filesystem files. It constructs an independently closed SlotContents value from that store.

Content encoding and upload

Ordinary objects and filesystem files are encoded using their repository envelopes. For each value, the publisher builds a Zstandard representation in a temporary file and compares its complete encoded size with the identity representation. It stores Zstandard only when it is smaller; otherwise it stores identity encoding.

Encoding and upload are streaming operations. Input or encoded content is not loaded into memory in proportion to its size.

The command encodes content in bounded temporary batches and overlaps a bounded number of S3 uploads. It never prepares the encoded form of the entire store in temporary storage at once.

Repository keys are immutable. The publisher does not replace an existing o/<ObjectHash>, f/<FsFileHash>, or immutable metadata key. A missing value is uploaded with a conditional create operation. A successful retry reuses an already present key. S3 single-object and multipart uploads must be completed before the value is considered available; incomplete multipart uploads are not repository objects.

The publisher applies the HTTP representation metadata required by the wire format, including immutable cache policy and the exact media type.

`prepare`

prepare runs on the build server. Its conceptual interface is:

bobr-repo prepare \
    --store STORE \
    --repository s3://BUCKET/PREFIX \
    --master-url https://repo.example.org/master \
    --trusted-key PUBLIC-KEY ... \
    [--data-base-url HTTPS-URL] \
    [--cache DIR] \
    [--ca-bundle CA-BUNDLE.pem] \
    [--quiet] \
    [--append | --add-slot | --rotate] \
    [--retention DURATION] \
    --output candidate-master.cbor

--append is the default for an existing repository. The three slot actions are mutually exclusive.

--master-url is the stable public URL by which clients identify the repository. prepare still reads the authoritative master directly through the S3 API; the URL selects the same RepositoryCache namespace that clients and status use. --cache enables that persistent shared cache. Without it, prepare uses a temporary cache.

Empty repository

If /master is absent, the default operation creates one current slot with serial = 1, populated exactly from the input store. The payload has previous_master_hash = null. --data-base-url is required. --add-slot and --rotate are rejected because there is no existing slot to modify.

Append

Append replaces the active current state with the union of its contents and the input store:

The old active state becomes retained until the retention deadline.
Its mappings and content sets are merged with the incoming store.
Incoming mapping candidates take priority over older candidates for the same build or reuse key.
The merged replacement receives serial = max(serial) + 1 and becomes the active current state.

The number of current slots does not change.

Before uploading content or allocating a serial, prepare --append encodes the merged active state and compares its build-index, reuse-index, ordinary-object-list, and filesystem-file-list hashes with the authenticated active slot. If all four hashes match and no expired retained state needs to be pruned, the operation is unchanged: it performs no S3 writes, allocates no serial, and leaves the candidate file untouched.

An expired retained state is a logical change even when the active slot is unchanged. In that case prepare produces a prune-only candidate: it removes the expired state while preserving the active slot and its serial. All immutable values referenced by this candidate already exist, so no content or metadata upload is needed.

Add slot

--add-slot appends a fresh current state containing exactly the input store. No existing current state is retired. The old active state remains current and becomes sealed because the new state has the greatest serial. The number of current slots increases by one.

Rotate

--rotate retires the current state with the smallest serial and appends a fresh current state containing exactly the input store. Other current states remain sealed. The number of current slots does not change.

Retention

The default retention duration is 24 hours. --retention overrides it for one candidate. Append applies the deadline to the replaced active state; rotation applies it to the retired oldest current state. Add-slot retires nothing and rejects an explicitly supplied retention duration.

At the start of every preparation, retained entries whose retain_until deadline has passed are omitted from the candidate. They remain live in the authoritative repository until that candidate is signed and published.

Preparation result

prepare reports exactly one compact JSON value on standard output:

{"result":"candidate"}

or:

{"result":"unchanged"}

Both are successful results. Human-readable progress remains on standard error. --quiet suppresses the live progress display without changing the JSON result or error diagnostics. unchanged is possible only for append to an existing repository; initial publication, add-slot, rotate, and prune-only transitions always return candidate.

For a changed slot, prepare uploads all missing content and its four immutable metadata values before writing the output file. It never writes /master. A prune-only transition needs no immutable uploads.

For candidate, the output file is the exact deterministic-CBOR master payload defined by master-payload.cddl, without a COSE envelope or signature. There is no separate candidate wire format. Its previous_master_hash is:

null for the first publication;
otherwise SHA-256 of the exact tagged COSE bytes fetched as the current /master during preparation.

The candidate contains no secret. It can be copied to the trusted workstation over an ordinary authenticated transport such as SSH.

The candidate is written to a temporary file in the destination directory, synced, and atomically renamed over the requested path only after every prior step has succeeded. An error or unchanged result leaves any existing file at that path byte-for-byte untouched.

Running prepare again is safe. It may reuse immutable values uploaded by an interrupted or superseded preparation. A newer changed preparation from the same current master supersedes an older candidate; only one of them can pass the predecessor check after either is published.

`publish`

publish runs on the trusted workstation:

bobr-repo publish \
    --candidate candidate-master.cbor \
    --repository s3://BUCKET/PREFIX \
    --signing-key PRIVATE-KEY \
    [--trusted-key PUBLIC-KEY ...] \
    [--ca-bundle CA-BUNDLE.pem] \
    [--yes]

The command performs these steps in order:

Read the candidate under the master format size limit.
Require its exact deterministic-CBOR representation and validate all master payload constraints.
Fetch /master directly through the S3 API, bypassing public HTTP caches.
For an existing repository, authenticate the current COSE master and require the SHA-256 of its exact bytes to equal previous_master_hash.
For the first publication, require previous_master_hash = null and require /master not to exist.
Display a structured comparison of current and candidate slot states, including data_base_url, added states, newly retained states, pruned states, serials, and deadlines.
Obtain explicit user confirmation unless --yes was supplied.
Derive the protected COSE kid, sign the exact candidate payload with Ed25519, and encode the tagged deterministic COSE_Sign1 master.
Conditionally create or replace /master using the S3 entity validator obtained in step 3. Fail if the object changed in the meantime.
Read the stored object back and require it to be byte-identical to the signed value.

The command does not download and independently verify all immutable content. The signed payload authenticates metadata hashes, and clients validate the complete hash chain lazily when they use it. Semantic verification of build results would require an independent build and is outside this command.

Interactive publication writes its comparison and prompt to the terminal. When standard input is not a terminal, omission of --yes is an error. The flag suppresses only the prompt; it never suppresses format, predecessor, signature, or conditional-write checks.

The private key is never copied to the build server, embedded in the candidate, written to the repository, or included in logs.

Publication and garbage-collection serialization

The repository has one logical writer. prepare, publish, and gc must be serialized as one administrative workflow even though signing occurs on a different machine.

In particular, gc must not run between prepare and either publication or explicit abandonment of its candidate. Content uploaded by prepare is not a live root until the candidate becomes /master; a concurrent collection may therefore delete it. No later candidate may be published after a collection that could have removed its inputs.

The first implementation enforces this operationally and through narrowly scoped credentials; it does not introduce repository leases or mutable pending publication records.

`status`

status is a read-only administrative command. Its standard output is one pretty-printed JSON document. Progress and diagnostics go to standard error, so standard output remains machine-readable. --compact selects compact JSON.

Logical status

bobr-repo status \
    --master-url https://repo.example.org/master \
    --trusted-key PUBLIC-KEY ... \
    [--cache DIR] \
    [--ca-bundle CA-BUNDLE.pem] \
    [--quiet] \
    [--compact]

The command authenticates /master and loads all four metadata files referenced by every current and retained state: build index, reuse index, object list, and filesystem-file list. The current implementation also loads mapping indexes even though content counts primarily require the lists. With --cache, unchanged metadata is read and verified locally; without it, metadata is downloaded again on each invocation. No o/* or f/* payloads are downloaded. It reports:

state = "ready", the number of current slots, and the active slot serial;
the master URL, data base URL, signing key identifier, and master hash;
active, sealed, and retained states with their serials and deadlines;
object and filesystem-file references per state;
unique repository content hashes;
hashes shared by several current or retained states;
hashes exclusive to each state;
expired retained states which a future preparation may prune.

This mode does not list S3 and cannot report encoded byte sizes or unreferenced bucket objects. Consequently, the top-level active_slot.content_bytes field is null.

Shared metadata cache

--cache DIR uses DIR as a RepositoryCache root. The directory layout, master-URL namespace, ETag handling, immutable metadata paths, and validation rules are exactly those used by the Bobr remote-repository reader. The Bobr client can therefore use the same directory directly; status does not create a CLI-specific cache format.

The cache is untrusted. The master is revalidated and signature-checked, and immutable metadata is rehashed before use. Without --cache, status uses a temporary cache and removes it on successful or failed exit.

Physical storage scan

bobr-repo status \
    --master-url https://repo.example.org/master \
    --trusted-key PUBLIC-KEY ... \
    --repository s3://BUCKET/PREFIX \
    --scan-storage \
    [--cache DIR] \
    [--ca-bundle CA-BUNDLE.pem] \
    [--quiet]

This mode treats authenticated S3 as the authoritative administrative view. It reads and verifies master directly through S3, fetches the immutable metadata named by that master through the S3 transport, and lists the recognized S3 namespaces. A public HTTP cache therefore cannot make publication decisions from a stale or missing master.

S3 listing returns encoded object sizes, so the command does not download every o/* and f/* value or issue a separate HEAD request for every key.

The stable top-level policy fields for a populated repository are:

{
  "state": "ready",
  "current_slots": 3,
  "active_slot": {
    "serial": 17,
    "content_bytes": 184532000000
  }
}

current_slots excludes retained slots. content_bytes is the sum of the encoded S3 sizes of the distinct o/* and f/* keys advertised by the active slot. It excludes b/*, r/*, lo/*, lf/*, and master. Content shared by several slots is counted in the size of every slot that advertises it; this is a policy measure, not exclusive physical ownership.

An accessible bucket without master is a successful empty result:

{"state":"empty","current_slots":0,"active_slot":null}

A definitive S3 NoSuchBucket is also a successful typed result:

{"state":"missing","current_slots":0,"active_slot":null}

The publication wrapper decides whether missing permits bucket creation. Authorization failures, TLS failures, timeouts, and other ambiguous errors are not converted to missing.

The JSON result additionally reports:

total physical bytes and key counts;
bytes split among ordinary objects, filesystem files, indexes, lists, and master;
referenced bytes per slot, which count shared content in every referencing slot;
bytes exclusive to each slot;
currently unreferenced immutable keys and bytes;
total bytes that become reclaimable after pruning all expired retained states;
per-slot encoded content bytes used by publication policy.

If an authenticated master advertises an immutable metadata or content key which the S3 scan cannot find, status fails instead of reporting a truncated size. A master change during the scan causes one complete retry; another change is an error rather than a mixed snapshot.

Per-slot size is never presented as one ambiguous number. The output names referenced_bytes, exclusive_bytes, and reclaimable_bytes_if_pruned_alone separately. The repository-wide reclaimable_after_expired_prune_bytes also counts data shared exclusively by several expired states, which cannot truthfully be assigned to any one state.

The scan is linear in the number of S3 keys and can require one LIST request per service page, commonly one thousand keys. It is intended for occasional administration, not a live monitoring loop. --cache caches authenticated master metadata but does not turn an old S3 listing into authoritative current storage state.

`gc`

gc runs with the administrative content-delete identity:

bobr-repo gc \
    --repository s3://BUCKET/PREFIX \
    --master-url https://repo.example.org/master \
    --trusted-key PUBLIC-KEY ... \
    [--cache DIR] \
    [--ca-bundle CA-BUNDLE.pem] \
    [--dry-run]

It never modifies or signs /master. Expired retained entries still present in the authoritative master remain live; they are removed only by a separately prepared and published master.

As with prepare, --master-url selects the client-compatible cache namespace while the authoritative master is fetched directly through S3. --cache persists authenticated metadata; without it, gc uses a temporary cache.

The command:

Fetches and authenticates the current /master directly through S3.
Fetches and hash-verifies every build index, reuse index, object list, and filesystem-file list referenced by every current and retained state.
Computes the complete live set for each recognized immutable namespace.
Lists those namespaces in S3.
Produces a deletion plan containing only keys absent from the corresponding live set.
Prints counts and encoded bytes by namespace.
Immediately before mutation, requires the authoritative /master to be unchanged since the live-set analysis.
Deletes the planned keys unless --dry-run is set.

Failure to fetch or validate any live metadata aborts before deletion. Keys outside b/, r/, lo/, lf/, o/, and f/ are never deleted. Interrupted deletion is safe to repeat because every retry recomputes the live sets from a freshly fetched and verified master.

--dry-run performs the same authenticated traversal and S3 listing as a real collection. It differs only by not issuing DELETE operations.

Exit and output rules

Every subcommand returns nonzero on validation, authentication, S3, local I/O, or conditional-publication failure. It does not downgrade a partial upload, stale candidate, missing advertised object, or incomplete garbage-collection analysis to a warning.

Machine-readable results are written to standard output. Human progress, retry notices, prompts, and diagnostics are written to standard error. Secret keys, S3 secrets, authorization headers, and private-key passphrases are never logged.

Interactive prepare and status commands keep one live progress line on a terminal. Redirected standard error receives phase transitions without terminal control sequences. --quiet disables this progress output while preserving machine-readable results, final informational messages, and errors.

All mutating commands support interruption. An interrupted prepare may leave unreferenced immutable keys, and an interrupted gc may leave unreachable keys. Both states are safe. An interrupted publish leaves either the old master or the complete conditionally written new master because S3 object replacement is atomic.

Keyboard shortcuts

Bobr