pkg-provenance(1)

NAME

pkg-provenance - verify, reproduce, or fetch the sources of a published build

SYNOPSIS

pkg provenance verify target [--trust-key HEX]...

pkg provenance rebuild target [--trust-key HEX]... [--recipe DIR] [--work DIR] [--keep] [--license-corpus DIR]

pkg provenance download-sources target --out DIR [--trust-key HEX]...

DESCRIPTION

A signature on a package tells you who published it. It doesn't tell you what it was built from, or whether anybody could build it again. Those are the questions that matter when you're auditing a system, and this command answers them.

Every published build carries a signed buildroot-attestation record. It pins everything that went into the build: the exact buildroot inputs, the upstream sources and their signature attestations, the recipe, and the outputs it produced. The three subcommands each read that record:

• verify checks that the record is genuine, and that everything it names is still there to rebuild from.
• rebuild actually rebuilds it, offline, and compares the result byte for byte.
• download-sources hands you the upstream source archives it was built from.

What can target be?

A package NAME, or a PATH to a published manifest (a pkg-meta in a repository tree).

A bare word is treated as an installed NAME first, so a stray file called attr in the current directory never shadows the installed attr package. A word with a / in it that names an existing file is a PATH. A bare word that isn't installed falls back to a file of that name in the current directory, and then to the build the configured repositories publish under that name: the one pkg install would take. So you can audit or rebuild a package without installing it.

For a NAME, the manifest of that build is looked for under --repo first. That's where it is on the build host, or on a machine that mirrors the repository. If it isn't there, pkg asks each configured repository in turn and fetches the manifest, the record and every input it reads into a local cache as it goes. Plain HTTP is fine for that, because none of the trust comes from the transport; see below.

Either way, a provenance.toml stub beside the manifest points at the signed record. The record names its inputs by repository name, and each one is found next to the target's repository: the one that states that name in its signed pin file, or else a directory of that name. Fetched, an input comes from the configured repository of that name, or else from beside the target's repository. A repository nothing configures and that isn't served there -- the Buildroot-N.R floor a Base package was built on, say -- needs --with-extra-repo or --input-base.

Why trust the record?

The manifest is the anchor, so it's checked before the record is even read. Its own signature has to verify. For an installed NAME, it also has to be byte-identical to the manifest installed here. For a NAME that isn't installed, it has to carry the signature the repository's verified index lists for that build. For a PATH, its signer has to be a key some installed package is signed with, or one you named with --trust-key.

Then the record has to be signed by that same key. A repository signs its packages and their records with one key, so a record from anyone else is refused. And the record has to list this exact manifest among its signed outputs. The stub is only a pointer, and a pointer alone could aim at someone else's record.

Everything else -- every input manifest, every payload, every source archive -- is checked against a hash in that record.

COMMANDS

verify

Checks the record without rebuilding anything:

• the record's signature, and who signed it;
• that the record's outputs bind to this package;
• that every pinned input is present in the repository and hash-intact. That's reproduction-completeness: whether an offline rebuild is possible at all;
• each upstream source, with its pin-time signature attestation: which key verified it, and when it was signed;
• which builder, at which version, made it, and which probe policy -- the flag contract -- it was built under.

For gzip, that looks like this:

verified gzip/bin 1.13-4.x86_64 record 9f2c1a... signed_by 3a7f...c2e1 buildroot Buildroot-3.3 output bound yes inputs 214 present + hash-intact sources 1 (1 signature-verified) main b41c8e... gpg 8A1F9A6C (created 2023-08-19) builder ur v0.3.1-42-gda5cbbe3 probe policy 5d0e7a... (policies/)

A source with no attestation says unsigned instead. That's a fact about the upstream project, recorded when the source was pinned. It isn't a verification failure.

Older records carry less. A record from before schema 6 says unknown for the builder and unrecorded for the probe policy, and a schema-6 record with no pinned policy says it was built with the default of its time, which can't be checked anymore.

When inputs are missing or damaged, you get every problem at once (the first 20 of them, anyway), each naming the repository and the input:

reproduction-INCOMPLETE: 2 problem(s) across 214 input(s):

rebuild

Reproduces the build offline, and diffs the result against the record.

It materializes the record's pinned input sysroot by content address, restores the probe policy the build ran under, seeds the sources from the repository's sources/ store so nothing is fetched from upstream, re-runs the recipe build in the sandbox, and compares the rebuilt payload hashes against the record's signed outputs.

If the record pins a probe policy this pkg doesn't carry, and the repository's policies/ store doesn't have it either, the rebuild refuses. Building with different compiler flags than the record attests would only prove that different flags give different bytes.

Each output lands in one of five buckets:

ok

Rebuilt byte-identically.

MISSING

The recipe didn't produce this subpackage again.

EXTRA

Rebuilt, but not among the attested outputs.

UNVERIFIED

The payload matched, but the published manifest is missing.

DIVERGED

Rebuilt differently. The delta follows, with - for the attested artifact and + for this rebuild.

A clean run looks like this:

rebuild REPRODUCED gzip-1.13 rel 4 (full-manifest) record 9f2c1a... signed_by 3a7f...c2e1 buildroot Buildroot-3.3 outputs 4 matched, 0 diverged, 0 missing, 0 extra, 0 unverified (of 4) ok gzip/bin ok gzip/debug

Anything other than every output ok is a non-zero exit. That's the point: this is a gate, not a report.

Under --json the same report goes to stderr, after the build log, so a captured stderr reads like an interactive run. stdout holds only the JSON.

Rebuilding needs a sandbox with user namespaces. On a kernel without CONFIG_USER_NS, the sandbox won't start.

download-sources

Downloads the upstream source archives the package was built from. They're found through the signed record, fetched from the repository's sources/ store, and checked against their sha256 before they're written.

downloaded 2 source(s) to /tmp/gzip-src main b41c8e... -> /tmp/gzip-src/main patches 7d02f9... -> /tmp/gzip-src/patches

OPTIONS

--trust-key HEX

All three subcommands. Can be repeated. Another key a manifest given by PATH may be signed with, on top of the keys installed packages are signed with. The record still has to be signed by the same key as its manifest.

--input-base URL

verify and rebuild. Where to fetch inputs from a repository that no configured repository names: URL/<repository name>/.... It names a location, not a trust root; every input is still checked against the signed record.

--with-extra-repo FILE

All three subcommands. Can be repeated. A repository drop-in, in the repos.d format, added below the configured repositories for this one command. Its enabled key is ignored. Use it to point at a buildroot floor, or at another distribution's repositories, without configuring them on this machine. A floor's drop-in should set provenance_only = true (see pkg-repos.d(5)), so no other command can install from it.

rebuild

--recipe DIR

A recipe tree (a directory holding recipe.toml and its sidecars) whose name, version and release match the record. Optional for a schema-5 or newer record, which pins the recipe by content address and reads it from the repository's recipes/ store; this flag overrides that. Required for older records, which carry no recipe bundle.

--work DIR

Scratch directory for the materialized sysroot, the source cache and the build tree. A directory you name here has to be new, empty, or one pkg made on an earlier run, so it can't wipe out something of yours. The default is a per-record directory under $XDG_CACHE_HOME/pkg/rebuild (else ~/.cache/pkg/rebuild, and /var/cache/pkg/rebuild for root), removed on exit. It's deliberately never /tmp, which is often a small tmpfs that a full sysroot would overflow. Point it at a big disk for big builds.

--keep

Keep the scratch tree after the run so you can look at it, instead of removing it. Its path is printed.

--license-corpus DIR

A fallback SPDX reference-text corpus, for schema-1 records from before license references were pinned. Schema-2 and later records carry the license texts they used and build the corpus from the repository themselves, ignoring this flag.

download-sources

--out DIR

Required. The directory to write into, created if it doesn't exist. Each archive is written under its recipe source name, or under its sha256 when two names collide or a name isn't safe as a file name.

FILES

$XDG_CACHE_HOME/pkg/provenance/

Manifests, records and inputs fetched from a configured repository, laid out like the repository itself. Falls back to ~/.cache/pkg/provenance/; root always uses /var/cache/pkg/provenance/. It's a cache: deleting it costs a re-fetch, nothing more.

EXAMPLES

Auditing something installed

pkg provenance verify openssl

Auditing a published manifest by path

pkg provenance verify \ /data/repo/Common/pkgs/gzip/bin/1.13/4/x86_64/pkg-meta

Which key signed the record

pkg --json provenance verify openssl | jq -r .signed_by

Which buildroot it came from

pkg --json provenance verify openssl | jq -r .buildroot_repo

Are the upstream sources signature-verified

pkg --json provenance verify openssl \ | jq '{total: .sources_total, verified: .sources_signature_verified}'

And which ones aren't:

pkg --json provenance verify openssl \ | jq -r '.sources[] | select(.verified_by == "") | .name'

Auditing an entire set

for p in $(pkg --json set show +console | jq -r '.expanded[]'); do pkg --json provenance verify "$p" 2>/dev/null \ | jq -r '"(.meta)\t(.signed_by)"' || echo "NO RECORD $p" done

Getting the sources

pkg provenance download-sources gzip --out /tmp/gzip-src ls -l /tmp/gzip-src

Sources for everything installed, for a compliance archive

pkg --json list | jq -r '.packages[].name' | sort -u \ | while read -r p; do pkg provenance download-sources "$p" --out "/srv/sources/$p" \ 2>/dev/null || echo "no record: $p" done

Reproducing a build

pkg provenance rebuild gzip

Reproducing a large build somewhere with room

pkg provenance rebuild linux-kernel-stable \ --work /var/tmp/rebuild --keep

Investigating a divergence

pkg provenance rebuild openssl --keep

The scratch tree's path is printed, and the diverged outputs are listed with their deltas, attested side as -, rebuilt side as +.

Reproducing with a local recipe tree

pkg provenance rebuild gzip --recipe /data/recipes/common/gzip

You need this for records older than schema 5. It's also a handy way to test a recipe change against a known-good record.

Scripting a reproducibility sweep

for p in $(pkg --json list | jq -r '.packages[].name' | sort -u); do if pkg --json provenance rebuild "$p" > "/tmp/rb-$p.json" 2>/dev/null; then echo "REPRODUCED $p" else echo "DIVERGED $p" jq -r '.diverged[].output' "/tmp/rb-$p.json" 2>/dev/null fi done

The complete audit of one package

pkg verify-sig /data/repo/Common/pkgs/gzip/bin/1.13/4/x86_64/pkg-meta pkg provenance verify gzip pkg provenance rebuild gzip pkg verify gzip

In order: the manifest is intact; the record is signed, bound to its outputs and its inputs are complete; the build reproduces byte for byte from those inputs; and the files on this machine match the manifest.

JSON

verify emits meta, record, signed_by, output_bound, inputs_total, inputs_verified, buildroot_repo, sources_total, sources_signature_verified, a sources array of {name, sha256, verified_by, key_fpr, sig_created}, and builder, builder_version and probe_policy_sha256.

rebuild emits reproduced, verdict, meta, record, signed_by, outputs_total, matched, diverged (each with its output and delta), missing, extra and unverified.

download-sources emits meta, record, count and a sources array of {name, sha256, path}.

EXIT STATUS

verify exits 0 when the record verifies and every input is present and hash-intact. rebuild exits 0 only when every output reproduced; any divergence, missing, extra or unverified output is a non-zero exit. download-sources exits 0 when every archive was fetched and sha-verified.

SEE ALSO

pkg(1), pkg-build(1), pkg-verify-sig(1), pkg-verify(1), pkg-info(1), pkg-repository-layout(7), pkg-chain(7)

docs/design/buildroot-attestation.md, docs/design/upstream-signature-attestation.md, docs/design/rebuilds.md, docs/design/sbom.md, docs/design/trust.md