The Ur Project

Documenting the craft of stacking things so they remain stacked. For a very long time.

Creating images

Every image on the Download page is built from a composition: a composition.toml file that names the repositories to draw from, the packages that go in, and the shape of the result. The compositions we publish our own images from are in the compositions repository, and anyone can build them with pkg compose.

This guide builds the live USB image, from our own composition, against the official repositories. You don't need a build system, and you don't need root. Composing doesn't change anything on the system you run it on.

What you need

  • A Sandcastle system. pkg is already on it, and so are the repository roots in /etc/pkg that the composition starts its trust from.

  • git, and for the later steps jq and qemu:

    $ sudo pkg install git jq qemu
  • About 6 GiB of free disk space, and not on a tmpfs: the uncompressed tree alone is 3 GiB. This guide uses /var/tmp.
  • A network connection. A compose downloads every package in the image, about 760 of them.

Get the compositions

$ git clone https://code.foo-projects.org/sofar/compositions
$ cd compositions
$ ls
README.md  Sandcastle-Base.keys  Sandcastle-Common.keys  baremetal
container-base  generic-cloud  generic-iso  live-usb  wsl-base

Each directory is one image. live-usb/ holds the composition itself and a seed/ directory, which is where the live user account comes from. The two .keys files list the keys the packages in each repository are signed with.

Read live-usb/composition.toml before you build it. It's heavily commented, and most of the comments are the reason a line is there.

Point it at the official repositories

The compositions are what our own build system composes from, so their repository URLs point at its local network. Point them at the public repositories instead:

$ sed -i 's|http://ur-build.lan|https://download.ur.foo-projects.org/repo|' \
    live-usb/composition.toml

That leaves the two repositories looking like this:

[[repos]]
name      = "Sandcastle-Base"
url       = "https://download.ur.foo-projects.org/repo/Sandcastle/Base"
tuf_root  = "/etc/pkg/sandcastle-base.root.json"
trust_key_file = "../Sandcastle-Base.keys"

[[repos]]
name      = "Sandcastle-Common"
url       = "https://download.ur.foo-projects.org/repo/Sandcastle/Common"
tuf_root  = "/etc/pkg/sandcastle-common.root.json"
trust_key_file = "../Sandcastle-Common.keys"
consumes  = ["Sandcastle-Base"]

The composition brings its own repositories: it doesn't use the ones your system is configured with. tuf_root anchors each one to the root your system already trusts, and trust_key_file names the keys every package has to be signed with. Paths in the file are relative to the file itself, which is why the keys are ../.

Look before you build

A dry run resolves everything and builds nothing:

$ pkg compose --composition live-usb/composition.toml --dry-run
composition live-usb (disk) -> live-usb/live-usb.qcow2
compose closure (760 package(s)) -> live-usb/live-usb.qcow2
  + NetworkManager             1.58.1-1      [/bin, /data, /lib]
  + accountsservice            26.27.3-2     [/bin, /data, /lib]
  + acl                        2.4.0-8       [/bin, /data, /lib]
...

The count is subpackages; the list is one row per package. This takes a few seconds, and it's worth doing every time you change the composition: a package name that doesn't resolve fails here, not 40 minutes in.

Note the file name it wants to write: live-usb.qcow2. Without --output, every disk image is named .qcow2, but this one is format = "raw", written to a stick byte for byte. So name the output yourself.

Build it

$ pkg compose --composition live-usb/composition.toml \
    --output /var/tmp/live-usb/live-usb.img --sbom spdx,cyclonedx --licenses

--sbom and --licenses write the same reports that come with the images on the Download page.

This takes a while. Ours took 40 minutes, almost all of it spent downloading, and a compose downloads everything again each time, so check the composition with a dry run first. When it's done, it says what it wrote:

composed disk image: /var/tmp/live-usb/live-usb.img
  raw intermediate: /var/tmp/live-usb/live-usb.staging.raw
  sbom: /var/tmp/live-usb/live-usb.img.spdx.json
  sbom: /var/tmp/live-usb/live-usb.img.cdx.json
  licenses: /var/tmp/live-usb/live-usb.img.licenses.json

Look at what came out

$ cd /var/tmp/live-usb
$ ls -1
live-usb.img
live-usb.img.cdx.json
live-usb.img.licenses.json
live-usb.img.spdx.json
live-usb.staging
live-usb.staging.raw
  • live-usb.img is the image, 1.7 GiB.
  • live-usb.staging.raw is the raw image it was converted from. For a raw image the two are the same file, hard linked, so it takes no extra space.
  • live-usb.staging is the tree that went into the image, uncompressed. It's left in place so you can look at it. The next compose into the same place removes it first.
  • The .json files are the reports.

The disk

$ sfdisk -l live-usb.img
...
Device          Start     End Sectors Size Type
live-usb.img1    2048  534527  532480 260M EFI System
live-usb.img2  534528  567295   32768  16M Microsoft basic data
live-usb.img3  567296 3461119 2893824 1.4G Linux root (x86-64)
live-usb.img4 3461120 3592191  131072  64M Linux filesystem

From the top: the EFI system partition with systemd-boot and the kernel; the cidata seed partition with the files from seed/; the root, a compressed read-only erofs; and the persistence partition, where whatever you change on the running stick is kept. It's only 64 MiB in the image, and grows to fill the stick on its first boot.

The packages

The staging tree carries its own package database, so pkg can look inside it the same way it looks at a running system:

$ pkg --state-root live-usb.staging/var/lib/pkg list --collapse main
NetworkManager             1.58.1-1      Sandcastle-Common  [/bin, /data, /lib]
accountsservice            26.27.3-2     Sandcastle-Common  [/bin, /data, /lib]
acl                        2.4.0-8       Sandcastle-Base    [/bin, /data, /lib]
...

And check the files on disk against their signed manifests:

$ pkg --state-root live-usb.staging/var/lib/pkg verify bash --root live-usb.staging

The reports

The SBOMs list every subpackage in the image, in SPDX and CycloneDX:

$ jq '.components | length' live-usb.img.cdx.json
760

The license report has each package's declared license, the files that back it up, and how much of the image couldn't be identified:

$ jq '.totals' live-usb.img.licenses.json
{
  "ids": 48,
  "packages": 760,
  "unidentified": 14,
  "without_text": 106
}

Boot it in a VM

QEMU can boot the image with UEFI firmware. -snapshot keeps the VM's writes out of the image file:

$ qemu-system-x86_64 -enable-kvm -machine q35 -cpu host -smp 4 -m 4G \
    -drive if=pflash,format=raw,readonly=on,file=/usr/share/qemu/edk2-x86_64-code.fd \
    -drive file=live-usb.img,format=raw,if=virtio -snapshot \
    -nic user,model=virtio-net-pci

You get the systemd-boot menu, and then the GNOME desktop, logged in as live-usb-user, with the installer pinned to the dash.

To put it on a stick, follow Writing it to a stick.

Make it your own

The composition is yours to change. Some places to start:

  • Add packages to [packages].needed, by name, as name/subpackage, or as a +set. Run the dry run again to see that they resolve.
  • Change the account in seed/user-data. The user name has to match graphical_autologin in the [guest] table, or GNOME logs in nobody.
  • Change the name and version in [composition], so your image can't be mistaken for ours.

Then compose again. Everything a composition can say is in the COMPOSITION DESCRIPTORS section of pkg-compose(1).

If you want images rebuilt by themselves whenever a package in them changes, that's what the build system does: see Deploying the build system.