pkg-chain(7)

NAME

pkg-chain - how pkg orders repositories and decides which build is current

DESCRIPTION

Most machines get their packages from more than one repository. There's a base layer, something stacked on top of it, maybe a local overlay with a couple of packages you built yourself. Sooner or later two of those carry the same name, and something has to decide which one you get.

Most package managers decide that by comparing version numbers, and pick the highest. pkg never picks a package that way, anywhere. Version comparison is how a stale copy in a forgotten repository quietly wins, and how a downgrade turns into a special case that needs its own flag. So I threw it out.

Instead, every pkg command that names a package resolves it against the chain: the configured repositories, arranged in a precedence order, highest first. Two rules govern it, and neither of them looks at a version number.

Position decides

The first repository in the chain that pins a name answers for that name. Full stop.

A pin is the repository saying "this build is the current one for this name". A repository that deleted a package simply has no pin for it, so it has no answer, and its archived copy can never win by sorting higher. That's what makes deletion in a higher repository mean something: when Base drops openssh, Base has no pin for it, and Common's pin is the pin.

What about a repository that publishes no pins at all, like a bare development directory? It gets asked only when no repository above it pins the name. Then its index has to hold exactly one build of that name. Several builds sitting side by side with nothing saying which one is current is an error, never a silent maximum. The error names the versions and tells you to pass name=version, or to publish the repository properly.

Shadowing is per name, not per subpackage

Every subpackage of a name comes from the one repository that wins that name. There's no falling through to a lower repository for a subpackage the winner doesn't have.

That sounds strict, and it is, for a good reason. Two repositories can publish the same name at the same version-release with different payloads. Common ships expect/debugsrc where Base doesn't, and their expect/bin payloads differ even at an identical version. Build a package out of pieces from both and you've made a machine that exists nowhere upstream.

Identity, not comparison

The pin is a named pointer, not a version floor. Convergence asks one question: is the installed artifact the artifact the pin names? It answers it by comparing payload digests, the sha256 of the bytes. That has three consequences:

Moving a package from Base to Common is real work, even at the same version-release, because the payload digests differ.
A release bump that rebuilt byte-for-byte identical isn't work at all, and is left alone.
A downgrade needs no special case. The pin points where it points, and you follow it.

ORDERING

Who decides the order?

The repositories themselves, mostly. A repository knows what it was built on top of, so it says so in its signed metadata, and that beats anything the local configuration guesses. The configured repositories are ordered by three rules, applied in this order:

Attested consume edges win. A repository states its own position in its signed metadata/provisional_pins.json. Common attests that it consumes Base, so Common sits above Base, whatever the local configuration says. Attested edges that name a repository this machine doesn't configure (a build-only repository, say) are dropped, not treated as an error.
Configured consume edges order the repositories that attest nothing. See the consumes key in pkg-repos.d(5).
priority, highest first, orders the repositories that no edge relates. Equal priorities fall back to the repository name, so the order is always the same.

A consumed repository can never outrank its consumer, whatever the priorities say.

What gets refused

Some configurations are refused outright, instead of being ordered somehow:

a consumes edge naming a repository that isn't configured (a dangling consume);
a cycle in the consume edges;
two enabled drop-ins with the same name;
a drop-in with an empty name;
url without tuf_root, or tuf_root on a path repository;
url and path together, or neither.

A dangling edge doesn't blow up where you created it. It refuses the next pkg command instead, which is why pkg-repo(1) checks what the chain would look like after the change before it writes anything.

A chain that spans two product lines (streams) isn't refused, but pkg warns about it every time it builds the chain. Packages would resolve across builds that were never made or tested together, which you should only see while migrating from one stream to another.

WHERE PINS COME FROM

Local pins, above everything

Above every repository sit the local pins in /etc/pkg/pins.d, written by pkg-pin(1) and pkg-rollback(1), or by hand. A name pinned there resolves to that build first. That's how a rollback survives the next upgrade. See pkg-pins.d(5).

A repository's own pins

For a local path repository, pkg reads the current pointer from the first of these that exists:

metadata/pins.json -- the blessed pins.
metadata/provisional_pins.json -- the staged pins, for when nothing has been blessed yet.
metadata/targets.json -- the signed TUF target list, which already names one build per identifier (through metadata/current.json in a repository published since that split).

If none of them exist, it's the one-build-or-refuse rule from DESCRIPTION.

A remote url repository serves its staged pins, so pkg reads metadata/provisional_pins.json there, verified through the TUF chain like everything else. A remote repository that predates pin files falls back to the current set its signed targets name.

pkg upgrade --provisional deliberately converges a local repository onto provisional_pins.json, even when a blessed pins.json exists. It only changes which file a local repository is read from; a remote one is read from its staged pins either way. See pkg-upgrade(1).

Sets are the exception

Set membership comes from metadata/sets.json, and that one is unioned across the chain rather than shadowed. See pkg-sets(7).

BYPASSING THE CHAIN

Naming repositories on the command line with --repo-url and --tuf-root replaces the configured set entirely. That's a deliberate bypass, for testing. Each --repo-url is verified by the --tuf-root in the same position, highest precedence first. --repo alone points at a single local repository, and it's what a machine with nothing in repos.d falls back to.

EXAMPLES

Seeing the configured order

pkg repo list pkg repo list --all

This view is deliberately offline: it shows what repos.d describes. The runtime order can differ, because attested edges fetched from the repositories themselves outrank the configuration.

Seeing the runtime order

The chain is announced on stderr at -v:

pkg -v list --all | head -0 pkg -v upgrade --dry-run 2>&1 | grep chain

Which repository answers for a name

pkg list --all | grep '^openssh' pkg set show +console

pkg set show prints, per member, which repository contributed it and what it resolves to. It's the quickest way to watch the chain make a decision.

Diagnosing a held package

pkg upgrade --dry-run

A line reading "held (no repo in the chain pins these): ..." means exactly what it says: no repository in the chain answers for that name any more. Nothing gets removed and nothing gets guessed. Either a repository is disabled that shouldn't be, or the package really did leave the distribution.

Testing a staging repository without reconfiguring

pkg --repo-url http://staging:8082/Common \ --tuf-root /etc/pkg/common.root.json \ upgrade --dry-run