The Ur Project

Documenting the craft of stacking things so they remain stacked. For a very long time.

Architecture - Packages - part 7

Another somewhat short post today about packages. This is what we're calling the things that come out of the build - aggregations of build products.

The data and the meta

Ye olde RPM (and so is deb) is a format with both metadata and data contents in a single container. This is bad. The single most reason why this is bad is that for most of the time, packages do not change that much. Maybe the binary changes a few bytes, or a lot of bytes. But by and large, the amount of the package that changes in most releases is very small. We knew about this problem for a very long time, and the classic distros have just stuck with it. Of course, it works really well. It's just wasteful. This is why delta RPMs exist. This is why Clear Linux OS built a different update mechanism based on binary deltas. It's all optimizations for different problems. Each with its own architectural consequences.

In The Ur Project we don't do this anymore. The fundamental concept of a package that comes out of the build system is split into two different parts: First, the metadata. And second, the data.

The metadata

The metadata of a package is published as a signed manifest. It lists the contents of the package (which, if you've been reading the posts, is really per subpackage) and all the attributes of that package. So, it contains the usual things like the name, version, release, what the name of the subpackage is.

[package]
arch = "x86_64"
name = "zlib"
release = 13
subpkg = "lib"
version = "1.3.2"

[payload]
algorithm = "tar.zst"
sha256 = "f6836d2fda7c91daea52bfdb64d46a42531fa39af10d8b610d05838d06214068"
size = 62565

The top of the zlib/lib manifest: identity, and a reference to the payload

Then, it lists the sha256sum of the data payload. And it records its size. This reference is to the CAS blob. That means that the checksum value itself is its location on the repo server. It also means that if two packages share the same sha256sum, an update from one to the other is an effective NOP update. For instance, if you have a package that needed a small revision to a minor build issue, and the version number and binaries didn't actually change between the two builds, they would share the same data payload file in the CAS, and wouldn't be downloaded again if it already was installed.

zlib/man 1.3.2-4    fa15e872f29bfc37...
zlib/man 1.3.2-5    fa15e872f29bfc37...
zlib/man 1.3.2-6    fa15e872f29bfc37...
zlib/man 1.3.2-7    4f19e61d459a0af6...
zlib/man 1.3.2-8    4f19e61d459a0af6...
zlib/man 1.3.2-9    4f19e61d459a0af6...
zlib/man 1.3.2-10   4f19e61d459a0af6...
zlib/man 1.3.2-11   e8acfa6795311794...
zlib/man 1.3.2-12   e8acfa6795311794...
zlib/man 1.3.2-13   e8acfa6795311794...

zlib/man was built 10 times, but only 3 distinct payloads exist in the CAS

The metadata records the license of the source code, in full SPDX, as well as captured license files from the install phase. And it records the human metadata stuff like the description and URL stuff.

And then it records every file in the data. For each file, it records the ownership, permission bits, path, size, type, and the sha256sum. With this data, the on-disk files can be fully verified. One thing that is missing is timestamps: No atime/mtime/ctime timestamps are recorded. The reason for this is so that the manifests don't change with every build, and we can avoid all the pitfalls of trusting file timestamps for verification. If we need to verify data, we really need to read the contents to be sure.

[[files]]
build_id = "e60775932429e0d7536f677780b7483cf86204a6"
gid = 0
mode = "0755"
path = "/usr/lib/libz.so.1.3.2"
sha256 = "eab646c258a505ed2c9c87e557eb99a3af5b24dab7866b4c4b39dd83deb85d34"
size = 108656
type = "regular"
uid = 0

A single file entry in the manifest - there are no timestamps

$ pkg verify gzip
gzip/bin
  ok    f /usr/bin/gzip
  ok    l /usr/bin/gunzip
  FAIL  sha256 /usr/bin/zcat
  FAIL  missing /usr/share/man/man1/gzip.1

pkg verify checks the files on disk against the manifest

At the end of every metadata file there's an embedded signature over the manifest itself, sans the signature part. This is the part that enacts the full circle trust implementation of the TUF - The Update Framework, and it ensures the updates are what they claim to be - verifiably so.

[signature]
algorithm = "ed25519"
public_key = "9c8084eb4024ab2dc503a3d5b2098095a76fd2093df045dc20ca3073b16ef704"
value = "c67bb9e6bf6ff07315dff09fe347ace1d8c40717e254468cdc1aea8709e8...4970b"

The signature embedded at the end of every manifest

The data

The package format that the build system generates is a pure tar. It's not entirely that simple though. We use the GNU Tar extensions to handle longer filenames. And we compress with zstd. This is for client-side compression speed. You may remember that in Clear Linux OS we did something different: we compressed every package with every compression format, and then picked the smallest size, which then got published. The downside of that is that it throws a lot of compression CPU time at the problem (but hey, it's only server side once) but second is that xz tended to win a lot, and xz decompression can be slow. So this time I'm opting for zstd everywhere. It's a deliberate choice to optimize for lower CPU time on the client.

Inside the tar files is exactly what you expect - /-indexed files with content. Nothing more.

$ zstd -dc f6836d2fda7c91daea52bfdb64d46a42531fa39af10d8b610d05838d06214068 | tar -tv
lrwxrwxrwx 0/0        0 1970-01-01 00:00 usr/lib/libz.so.1 -> libz.so.1.3.2
-rwxr-xr-x 0/0   108656 1970-01-01 00:00 usr/lib/libz.so.1.3.2
drwxr-xr-x 0/0        0 1970-01-01 00:00 usr/share/pkg-licenses/zlib
-rw-r--r-- 0/0     1002 1970-01-01 00:00 usr/share/pkg-licenses/zlib/LICENSE
drwxr-xr-x 0/0        0 1970-01-01 00:00 usr/share/pkg-licenses/zlib/contrib
drwxr-xr-x 0/0        0 1970-01-01 00:00 usr/share/pkg-licenses/zlib/contrib/minizip
-rw-r--r-- 0/0     3509 1970-01-01 00:00 usr/share/pkg-licenses/zlib/contrib/minizip/LICENSE.Info-Zip

The zlib/lib payload: a plain tarball, every mtime zeroed, all owned by root

Bundles?

Yes, you're right, there are also bundles. You're asking me why, aren't you?

There's a small amount of cases that warrant you walking down from one computer to another one with a USB stick to carry something small over. It's not because you particularly enjoy that walk-of-shame. It's because without that walk, that computer at the other end has no network connection and you messed up. And now you're stuck downloading something to bring it over and trying to install it so it can get back online without anyone else in the office noticing that the reason that there's no customer issues coming in is because everything is offline.

But this is super problematic if the repo layout forces you to download two separate things for each subpackage and you might need 20 subpackages. Thank goodness that with pkg on the target, and pkg on the source machine, this is reduced to a relatively simple problem.

pkg supports doing full offline upgrades because the tool allows you to export a pins file to a USB stick, take it to an online source machine, and fill the USB stick with an upgrade bundle generated based on the pins file you brought over. Bring the bundle file to the offline host and it applies the update from it as if it's online. You can manually create any bundle file you want by specifying the content you want in it, too. If you built a package locally, it too can be put in a bundle for distribution to friends or anywhere.

--sofar (aka Auke)