The Ur Project

Documenting the craft of stacking things so they remain stacked. For a very long time.

Where it's going

I've previously written about some basic ground rules that I set, and outlined some shapes. These are based on my personal experience and observations of users of the distributions that I worked on. The biggest user I watched most of the time would have been myself. I'm a big fan of using the fruits of my own labour, and a big critic as well. I've drawn conclusions based on my own usage patterns that were completely wrong. Fortunately, development can be iterative. Get it right and you will doubt yourself and look for things that are wrong. Get it wrong, and you will instantly know at a certain point. Do this long enough and sure enough at some point there's no obvious wrong things anymore.

I started mentioning my frustration at not being able to truly build a community around Clear Linux OS and most importantly because I felt that we were inhibiting users from making improvements. Hands down the number of user-provided patches for actual packaging content must have been at its peak a patch per month. This is just terrible when you think of what it means. There's no longevity. And worse, there was/is no reason for Intel to let someone else steward the project.

So up front I wanted to build something that puts sharing and inheritance of what people have built front and center. It must be easy to create, publish and use repositories, while still offering the same level of security. This means something specific however, a few levels down the build system stack. What it means is that the way that you build things isn't something that's necessarily part of the ... hmm I need a name for this - declaration of inputs to a package.

So the term I'm using is "recipe". It's like your grandmother's collection of cooking recipes: It lists the ingredients, and amounts, but it's largely devoid of detailed instructions on how to cook it. Perhaps it says to "whisk eggwhites". But it sure as never will say "mix flour out back on the porch while listening to the radio". The details of how the thing must be assembled are largely assumed to be generic cooking knowledge.

declarative, not instructions

So just like meson, and even systemd, the recipes used in the Ur Project are all looking like declarative stuff for "these are the ingredients" with maybe a sprinkle of dependencies and some exception cases. But most largely how these recipes are compiled, there's almost nothing.

The autobuilder component takes care of pretty much most of the building. It consists of a number of probes that will fire based on content, and the winning probe(s) will then be used to perform the build. There's no scripting involved here. It's all mostly native code with a very thin layer of configuration so that things like compiler flags for hardening and optimization levels can be inserted.

Here's an example of one of these recipes:

[recipe]
name = "zlib"
version = "1.3.2"
release = 13
description = "zlib -- general-purpose lossless compression library implementing the DEFLATE algorithm"
homepage = "https://www.zlib.net/"
license_hint = "Zlib"
summary = """
zlib is a general purpose lossless data-compression library implementing
the DEFLATE format used by gzip and PNG, designed to be free, portable
and never expanding the data.
"""

[[source]]
name = "zlib"
url = "https://zlib.net/zlib-1.3.2.tar.gz"
sha256 = "bb329a0a2cd0274d05519d61c667c062e06990d72e125ee2dfa8de64f0119d16"
unpack_to = "."

[probes]
disabled = [
    "cmake",
]

If you didn't recognize the format - it's TOML. As you can see there's a single source entry, the usual sha256sum, and some basics describing the project. There's also an exception to disable the cmake probe as zlib ships both a plain Makefile and even autoconf.

You can also see there's no %build. Nor is there a Makefile looking language. Nothing scripting really, except for a few install related things (but there really shouldn't be anyway).

The weird part of it is that this just works. There's no need to do weird build macros inside the recipe. If you want to do complex build things, you should patch the build code of the project you're working on. Not invent a secondary build layer. Or better yet, push it upstream.

registry

Terminology: Recipes live inside a registry. Packages live inside a repository. You publish the recipes as a git tree. It's mostly text files, and maybe some static assets. Easy to share, too.

The build system monitors the registry. Everything not yet built causes a job to spawn to build it. The builder handles dependencies and assembles the buildroot for the job, calls the autobuilder probes and decides a winner. The winner probe performs the configuration step, the build step, and the install step. And then the builder publishes the results into a repository.

repository

The repository contains metadata and data, itself. The metadata describes the package and the content. The data is the content. Metadata again is manifest data, TOML format. Entirely machine generated. It refers to the actual content by CAS - a sha256sum based content-addressed-storage. This deduplicates all the content, makes the filename in the repo identical to the checksum of said content, and that's just a very easy working model. Repos can be garbage collected in the long run, and the CAS can be pruned by discovering packages that no longer are desired to be hosted.

The repository contains more than a single CAS. It also contains source content in a CAS. This provides GPL compliance, but also allows for the consumer tool to rebuild any content - it's just there available as a cache for users to download, rebuild, or even just verify and read.

The repository also contains provenance records in a CAS. This allows packages to identify their origin and buildroot so that they can be replicated instantly by the consumer tooling, and verify that builds are reproducible, and the circumstances of the original build are exactly preserved. Similarly I bolted on a CAS for licensing references, and for actual recipe content.

The packages themselves are zst compressed tarballs. Nothing more. They do not contain metadata, at all. The reason for that is that tooling can separately read metadata and doesn't need to fetch or be forced to fetch the content until it's needed. This allows for tooling to do automated and cursory things like dependency resolution, creation of caches and indexes, and just work off references to content checksums only. And the checksum is the location of the content within the repository package content CAS anyway.

On top of that it gets us free deduplication at the subpackage content. If the only thing that changes in a binary is the executable, then the man and lang and all those other subpackages aren't another full download - the repo server only stores one copy in the data CAS.

The useful split

This may sound like it's difficult to work with air-gapped systems, but, the consumer side tooling can make bundles out of any number of packages. It will create a bundle (quite literally a tarball) of the needed CAS data for the given inputs. For truly airgapped systems there's a "manifest export" that allows you to generate a statefile of the airgapped system, take it to a system that is online, and create an upgrade bundle for the airgapped systems' specific upgrade requirements. You need exactly one USB stick and walk from the airgapped system to the online one, and back. Meanwhile the bundle created contains all the data verifying the origin of it all, and the offline system will verify it.

This elegant solution almost naturally rolled out of the choices to separate content from metadata. It's a different take on the problem that every content distribution mechanism has to reconcile. As I said before, I wanted to build something that tailors to no specific use case - but rather to all. By doing away with %build-style compilation scripts and rules like build instructions, making recipes largely declarative, and designing repos to be split out, I can do this. If I would have designed a system that glosses over details and is largely coarse-grained, I'd have little to offer when compositions would require fine-grained level of control.

--sofar (aka Auke)