The shape of a liquid

One more time, I'm going to write something about this project and what it is, what shape it has, and how you'll be able to use it, and even contribute.

The project consists of all the machinery needed to publish a Linux Distribution on a rolling (or not) cadence, and consume it. So, it is code that builds it, and it is code that consumes the built things.

The code that builds it and the code that consumes it are one and the same. They live in the same git repository. They use the same underlying internal libraries. They are tested using the same functional tests, but they have 2 shapes.

Everything is written in the Rust language. Nothing is in an interpreted language unless it has to be. The dependency resolution code that the build side uses is shared with the consumer. Repo parsing code similarly is shared. Manifest parsing is shared code, etc.. There are a few robustness benefits from having the consumer side implemented in rust as well. We're not dependent on interpreted language libraries at all. As a matter of fact, the consumer side can largely function even without a shell being available on the system. It only has a hard requirement on glibc and a few low level system libraries (bz2, gcc_s). This means we can use it inside container images without that container requiring a shell.

A secondary benefit is that the consumer side also can be used as composition tool. It isn't just "a package manager" - it also is an image building tool - it has full knowledge of how repositories look and can do everything the build system potentially could do. So it can validate all the ingested bytes, and it can even do a reproducible build - validating that the bytes on the local system are reproduced bit for bit exactly as how the remote build system that published them did.

Provider side

The build side of things looks a little how normal distributions are built. It's a scheduler that understands dependencies and can initiate builds for everything that needs building. If there are 5 things that need building, it builds whichever thing can be built now, before things that have unmet dependencies.

The builder understands that in order to build something, it must be able to assemble a build root for it. A build root might mean that it has to assemble things from multiple repositories, so, it understands relationships between multiple repositories and can figure out what exactly is needed from each one involved.

To make the relationship between repositories simple, the relationship is defined as "one repository can build upon another one". In essence, a tree model of repositories can be created.

The Base

At the root of all this sits the base of the tree structure. You have to start with something, so, in Ur Project terms, there is where the build root sits. The build root started out as the bootstrap. The seed of this came from Fedora 43, and I preserved the chain for historical purposes. This seed mechanism isn't hugely dependent on any particular origin, though. A few shell scripts convert whatever seed you desire to start with into the native package format and if all dependencies are met, you could bootstrap yourself just as easily.

bootstrapping becomes a periodical thing. In order to build the lowest levels of the distribution, one starts with gcc/glibc/binutils and a few packages. We create a special repository called Base and rotate it periodically when the base toolchain needs rotating. Either because it's old, or because we want the world to use new compiler features or toolchains. This rotation happens by publishing whatever is in Base into Buildroot, and do repeated build cycles where Buildroot-N+1 gets built by Buildroot-N, until we've reached a point where Buildroot-N+1 no longer differs in binary content from Buildroot-N, and for safety, that no longer differs from Buildroot-N-1. At that point we've proven stability, and promote the Buildroot to the new operational one. Usually it takes 4 or 5 rounds of building.

Common

The rest of the packages live in Common. I've stuffed everything in here for now, and it sits at about 1000 or so. This is where the bulk will live, for now. We could split out things in the future to multiple repositories, but there's not a significant need as of yet.

Repositories

A published repository is an organizational structure, published through e.g. an HTTP or HTTPS server that the consumer layer can use to create new compositions and maintain them (i.e. install/update/erase packages). It's just metadata and content in a specific structure on disk. Suitable for CDN or even S3 storage and publication.

Consumer side

The consumer side is code that understands the repository format and can make or maintain compositions. Ordinary per-package installations are one such composition method. But they could also be atomic installations, live USB images, container images or anything else. All of these "composition" methods share the same underlying architecture. This isn't super crazy, it's mostly what other distro's tend to do: anaconda just calls RPM under the hood to do the install. In Clear Linux OS - the outlier - this wasn't the case, due to the abstraction of bundles. We could build this again as a composition layer, but I don't think I'm at the point yet that it makes sense right now.

Wrapping it all up

In the end, this is looking like a classical build system that makes packages, publishes repositories, and a package management side that can use it to maintain the end-user side of things as the user wants sees fit. In a next post I'll start outlining how things are different.

--sofar (aka Auke)