pkg-repository-layout(7)

NAME

pkg-repository-layout - on-disk and URL layout of an Ur Project repository

DESCRIPTION

A repository is a directory tree. That's it. There's no database, no server process with an API, nothing that has to be running. The same tree is what a local path repository points at and what an HTTP server hands out as a url repository, so a mirror is a file copy and nothing more. Every path below is relative to the repository root.

Metadata and data live apart

A package isn't one file. It's split in two: a signed manifest that describes it (its identity, its dependencies, every file with its mode and checksum), and a payload, a plain zstd-compressed tarball holding the bytes. The manifest names the payload by its sha256.

That split is deliberate. Most of the time a new build of a package changes very little, and a manifest is a few kilobytes of text. A client can resolve dependencies, plan an upgrade and build indexes from manifests alone, and only fetch a payload when it actually needs those bytes.

Content-addressed stores

The payloads live in a content-addressed store: a blob's address is the sha256 of its contents. Know the checksum and you know where the blob is, with no index to search. Two builds that produced identical bytes share one blob, so a rebuild that changed nothing costs nothing to store and nothing to download. And the address is also the integrity check: fetch it, hash it, and you know whether it's right.

Every content-addressed store in a repository shards by the first two bytes of the hex digest:

Two levels, so that no single directory ends up holding every blob, which would be terrible for performance. A digest is always 64 lowercase hex characters. Anything else is refused, never turned into a malformed path.

TREES

metadata/

Signed repository metadata. See METADATA, below.

pkgs/

Package manifests, addressed by identity rather than by content:

pkgs/<name>/<subpkg>/<version>/<release>/<arch>/pkg-meta

Every published build has a directory here, superseded ones too. That's why name=version can still resolve an artifact that's no longer current.

A subpackage name may contain characters a path can't carry, so the directory component is the escaped form: lang/en@boldquot is stored as lang/en\x40boldquot. Note that the subpackage component can itself be more than one path segment deep (lang/de).

pkgs/.../provenance.toml

A small, unsigned pointer sitting next to each pkg-meta. It names the content address of that build's signed buildroot-attestation record. Because it shares the manifest's directory, it lives and dies with the package, and needs no index of its own. pkg-provenance(1) starts here.

data/

Payload blobs, content-addressed, with no suffix:

data/<sha[0:2]>/<sha[2:4]>/<sha>

That's the canonical layout. When reading a local directory, pkg also accepts a flat development layout, <dir>/<sha>.pkg-data, as a fallback. When a blob is missing, the error names the canonical path anyway, because that's the layout you should be pointing at.

provenance/

Signed buildroot-attestation records, content-addressed. These aren't TUF targets. Trust comes from each record's own signature, and its digest is both its address and its integrity check.

sources/

Upstream source archives, content-addressed: the exact tarballs each build consumed. This is what pkg provenance download-sources fetches from, and what makes an offline reproduction possible.

recipes/

Recipe bundles, content-addressed. A schema-5 or later provenance record pins the recipe by its content address, so pkg provenance rebuild can read the exact recipe tree out of the repository instead of being handed one.

license-refs/

SPDX reference texts, content-addressed. A record carries the license texts it used, so a rebuild assembles that corpus from the repository instead of depending on one you supply.

policies/

Build-flag policies (probes.toml), content-addressed, pinned by a schema-6 provenance record. The policy is a per-distribution build input that isn't in version control, so a bare hash in the record would point at nothing. A build that used the compiled-in default policy records no hash, and stores nothing here.

Why keep all of that?

Because with it, anyone can check that what was built is what they'd get themselves. The sources, the recipe, the build policy, and a record of exactly what was in the buildroot are all in the repository. Rebuild from those and the bytes should match. Without them, a reproducible build is a claim you can't test.

METADATA

How a client trusts any of this

A client never trusts a file because of where it came from. The metadata/ directory is a TUF (The Update Framework) chain. A small set of signed files vouch for each other, and in the end for every other file a client reads. A client refresh walks timestamp, then snapshot, then targets, and from there every other metadata file is checked against the hash targets.json gives for it.

metadata/1.root.json

The TUF root: the trust anchor. This is the file pkg repo add fetches and, with --fingerprint, checks. To actually authenticate anything, it has to be obtained out-of-band. Fetching it over the URL it's supposed to authenticate proves nothing. See pkg-repos.d(5).

metadata/timestamp.json, metadata/snapshot.json

The freshness end of the chain. These are tiny, and they're what a client reads to find out that nothing changed.

metadata/targets.json

The signed list of the repository's metadata files, with a hash for each: current.json, archive.json, pins.json, provisional_pins.json, sets.json and resolve-index. It stays small no matter how big the distribution gets. In a repository published before current.json existed, the per-manifest keys live in targets.json itself, and pkg still reads them from there.

metadata/current.json

The current set: the hash of every current build's pkg-meta, one build per identifier. When a repository publishes it, it is the whole answer. An empty one means the repository publishes nothing, not "look somewhere else".

A pkgs key here may carry payload_size and uncompressed_size in its custom object. They're copied in at publish time, so a catalog view can show sizes without parsing every manifest.

metadata/archive.json

The full publication history. When a build is superseded, its key moves out of the current set and into here. That keeps the current set bounded, and nothing is ever lost. pkg reads it only when a command needs history, which in practice means an explicit name=version spec.

metadata/pins.json

The blessed pins: for each package name, the version and release that's current. This is the named pointer the chain reads first, for a local repository.

metadata/provisional_pins.json

The staged pins, plus the repository's attested consumes edge. Read when no blessed pins.json has been published, read deliberately by pkg upgrade --provisional, and always the pin file pkg reads from a remote repository. The attested edge in it outranks the local consumes configuration when the chain is ordered.

metadata/sets.json

Published set definitions: name, description, members. Unioned across the chain rather than shadowed. See pkg-sets(7).

metadata/resolve-index

Every current manifest's resolver-relevant fields in one file, so a cold client fetches one file instead of one per manifest. It's advisory: it's derived from the pkg-meta files, which stay the source of truth, and a client that can't use it falls back to reading those.

Keys, both in the signed metadata and in URLs, mirror the tree exactly, with forward slashes and no leading slash. For instance pkgs/<name>/<subpkg>/<version>/<release>/<arch>/pkg-meta and data/<sha[0:2]>/<sha[2:4]>/<sha>.

READING ORDER

Given a package name, pkg resolves it like this:

Order the configured repositories into a chain (pkg-chain(7)).
For each repository, load its pins. A local repository: pins.json, else provisional_pins.json, else the current set targets.json points at. A remote repository: provisional_pins.json, else that current set.
Take the first repository in the chain that pins the name. That repository answers for every subpackage of the name.
Read the manifest at the pinned pkgs/... path.
Fetch the payload from data/..., by the digest the manifest carries.

Metadata first, bytes last. Nothing big gets downloaded until step five.

EXAMPLES

Anatomy of a local repository

ls /srv/repo/Common ls /srv/repo/Common/metadata

Finding one build's manifest by hand

find /srv/repo/Common/pkgs/gzip -name pkg-meta

The supported way:

pkg --repo /srv/repo/Common info gzip/bin

Reading a manifest that isn't in any repository

pkg info takes a path when the argument names a file that exists:

pkg info /srv/repo/Common/pkgs/gzip/bin/1.13/4/x86_64/pkg-meta

Locating a payload blob from a manifest

sha=$(pkg --repo /srv/repo/Common --json info gzip/bin \ | jq -r .payload.sha256) ls -l /srv/repo/Common/data/${sha:0:2}/${sha:2:2}/$sha

Serving a repository over HTTP

Any static file server rooted at the repository directory is a complete repository. Clients then use:

pkg --repo-url http://host:8082/Common \ --tuf-root /etc/pkg/common.root.json list --all

Mirroring

rsync -a --delete host::repo/Common/ /srv/mirror/Common/

Then configure it as a path repository, see pkg-repos.d(5). Boring, right? That's the point.

Following a build back to its sources

pkg provenance verify gzip pkg provenance download-sources gzip --out /tmp/gzip-src