The Ur Project

Documenting the craft of stacking things so they remain stacked. For a very long time.

Air-gapped system upgrades

Some systems never see a network. Maybe they shouldn't, maybe they can't, or maybe somebody broke the network config and now the box can't reach anything until it's fixed. That system still needs updates, and somebody is going to walk over with a USB stick.

That walk should happen exactly once in each direction. You carry a small file from the air-gapped system to a connected one, and you carry one bundle back. Everything in that bundle is verified on arrival, just like it would be online.

What you need

  • The air-gapped system, which this guide calls A.
  • A connected system, C, with pkg on it, following the same repositories as A. The bundle is built from what this system's repositories pin, and that is what A ends up at.
  • A USB stick, or anything else that carries a file across.

Nothing needs to be configured on A for this. There is no repository to add and no mirror to set up.

Export what the system runs

On A, write down what is installed:

$ pkg offline export-pins --output /mnt/usb/

This writes <hostname>.pins.toml: one entry per installed subpackage, with its name, version, release and payload digest, plus the chain the system follows and the keys that signed what it runs. It only reads, so it needs no privileges.

The hostname in the file name is only there so you can tell your USB sticks apart. Nothing parses it, and nothing compares it.

Walk the stick over to C.

Build the bundle

On C, build a bundle for exactly that system:

$ sudo pkg offline bundle-create --pins /mnt/usb/airgap-1.pins.toml \
    --tuf-dir /var/lib/pkg/repo/metadata --output-dir /mnt/usb/

This takes every installed subpackage in the pins file, works out what this system's chain pins for it, resolves the closure, and bundles only the packages A isn't already at. It's a delta, not a copy of the repository.

A package no repository in this chain pins is reported as held, and left out:

2 package(s) held -- no repo in this chain pins them: bzip2, tar

If A is already current, you get no bundle at all:

airgap-1 is already current against this chain; no bundle written

And if some of the bundle is signed by a key A has never installed from, it warns you now, so you find out before the walk rather than after it. See After a key rotation.

Walk the stick back to A.

Apply the bundle

On A, look at the plan first, then apply it:

$ sudo pkg offline upgrade /mnt/usb/airgap-1.pkgtar --dry-run
$ sudo pkg offline upgrade /mnt/usb/airgap-1.pkgtar

This is the same planner, the same confirmation prompt and the same triggers as a normal pkg upgrade. Every manifest in the bundle is verified before anything is applied, and nothing is extracted to disk first: the payloads are served straight from the bundle, so the system doesn't slowly fill up with superseded builds.

A package the bundle doesn't carry is left alone. Missing from a delta means unchanged.

As with any upgrade, you may need to reboot afterwards. See Basic system maintenance.

After a key rotation

A bundle on a USB stick has no TUF chain behind it, so the signature on each manifest is the only trust it carries. pkg offline upgrade trusts the keys that signed what is already installed, each for the package names it already signs on that system, and nothing else. The everyday update needs no flags.

That means that right after a signing key is rotated, the first bundle signed by the new key gets refused, every time. That's not a bug: an offline machine has no way of learning about a new key on its own. You tell it. The error names the flag that would accept it:

error: offline: openssl/lib 3.5.6-14 is signed by 5a31eaacac5c4b64, which signs
nothing installed here; pass --trust-key 5a31...2384 to accept it

Check the key against the published fingerprints, then pass it in. For more than one, keep them in a file, one per line (# comments are fine):

$ sudo pkg offline upgrade /mnt/usb/airgap-1.pkgtar --trust-key-file sandcastle.keys

Multi-key files need pkg 0.0.62 or newer. A key you pass this way is trusted for every package name, for that one run, so only pass keys you have checked.

You need the same flag for a repository the system never installed from, and for the very first bundle onto a machine with nothing installed.

Carrying a single package

To carry one package or a handful across, rather than upgrade the whole system, a plain bundle does the job. pkg bundle packs any set of packages into one signed .pkgtar, and Building local packages walks through making one and installing it.

For everything else, including --force, --no-verify and --keep-staged, see pkg-offline(1).