Architecture talk - dispatcher

Some big reveals coming

The project aims to create a community where developers can build any sort of Linux Distribution shape they want. The lowest resolution (pixel) of that shape is a package. Packages need to get built. This is done by dispatcher.

Dispatcher, or, dispatch

This is the name that I came up with to signal the function of this thing. It's the "messenger of stuff that needs to be done". The taxi command center. The thing that keeps all the drivers busy.

Functionally dispatcher looks a lot like koji. It's a job scheduler that understands resources and tries to chug away at getting things done. What it gets done is pretty much all package builds. It knows this because it watches registries (those are collections of recipes) and sees when there are unbuilt packages for any of the recipes, and makes sure they are built in order. Once built, it publishes the results, keeps a log, signs the manifests, and all that.

Repositories realize registries

A repository is the binary product of a fully compiled registry. In order to compile a recipe, you need to have a compiler, sometimes libraries, and more. Those already need to be compiled before they are usable. So, more than likely, you need a sort of repository to build the repository. If that sounds circular, it's because it is.

At first I designed this "graph" or "tree" of repos and registries in a clumsy TOML hard config type of way. But it was unintuitive and very hard to manipulate. And on top of that, for reasons I'll explain later, easy manipulation is going to be needed as branches and new repos are things we are using as tools to get certain things accomplished. So it has to be easy to understand the topology of these registries and repositories. Otherwise, it's just a mine field of problems.

Out of this came the "graph" topography view in the dispatcher web interface. I'll walk you through the elements in the graph (see image) so you understand the relationships.

• On the left - registries, containing recipes.
• On the right - repositories, containing packages.

All the 3 blocks on the left hand side are registries. base and buildroot are the same thing, but ignore buildroot for a bit. You can see the horizontal lines coming from the registries and go to the realizing repositories. So in the graph, you can see that Sandcastle-Common contains the packages built from recipes in common, and the same for Sandcastle-Base containing packages built from recipes in base.

In between Sandcastle-Base and Sandcastle-Common there's a floor arrow. It shows you that whatever compiles in the upper one of the two, inherits all the things provided in the lower one of the two. And, as you can see, it can stack, creating a long chain of repositories providing functionality to higher ones in the chain.

Historical Buildroots and the current one

None of this could have happened in a vacuum. You can't build a gcc package without gcc. I had to start from something. This is called "boot strapping" a distro. In Linux From Scratch, they'll show you how to do it. It's not rocket science. What I did was mechanize ingesting Fedora 43 packages and converting them to my desired native Ur Project package format, so that they could be replaced one at a time by natively built packages. I started with 250 or so RPMS, and converted them. These are now archived in Seed - the repo at the bottom right of the screen. And they will remain there.

Seed -> Buildroot-1.1

I then wrote the dispatcher and it built native binary packages for "my own recipes". This took a very long time, but after a while I had a full set of packages that would give me all I needed to drop Seed and rebuild all of Buildroot-1.1 against eachother. That broke everything. So more time was needed to refine the recipe format and the package format and the bootstrap extraction and ... it took an even longer time to finally get Buildroot-1.1 fully built.

And then again. A few times!

So to make sure we're not relying on something that we borrowed (and will drop) we do another full circular build. So we literally build buildroot again, but now using Buildroot-1.1 as the floor, and output into Buildroot-1.2. That showed some new problems and I had to go back to fixing Seed all over, wipe -1.1, build that again, and then -1.2 finally. This probably took 2 weeks of iteration just by itself, chasing bootstrap issues and building out dispatch as it grew along and gained implementation of new concepts and old tricks.

From Buildroot-1.2 we built Buildroot-1.3. From Buildroot-1.3 we built Buildroot-1.4. And then we compared Buildroot-1.3 to Buildroot-1.4 and it was identical. The manifests showed that the content of all the files were completely the same. Yzah! Eureka! We have a completely clean bootstrap!

So obviously we wiped all the Buildroot- repos and built the whole chain again. And then to be sure we added a .5 at the end, so that 1.3 == 1.4 == 1.5. And this has been the gate from there on. Each Buildroot generates its successor and once 3 successors are identical bit for bit, we have proven the bootstrap is done. Each subsequent recompile no longer will change the content.

And that's the new floor of Base. And Base becomes the floor of Common.

base is the working copy of buildroot. They are identical. buildroot is stable when a new Buildroot chain is being made. It contains tags indicating where Buildroot-X.Y was built from, recording the historical versions of the recipes, and the date that it was cut.

Retiring older Buildroots.

From the 1.x chain we started building out common and at some point, it became clear that we didn't need 150+ recipes in base, so I started to move stuff over that isn't needed for the bootstrap into common. This shrunk Base, and also, once gcc or something changes, it's a good idea to rebuild the buildroot out again. I've done this process now a few times, and each triple-equal winner is preserved for the next generation. The image shows which generations were winners. We throw away all the intermediates - they hold little to no value. Once stability is reached we can safely remove them knowing that we could repeat the process to recreate the intermediates, anyway.

This method provides us with a full historical lineage of our distribution content, back all the way to the original Seed content from our ancestor. The historical significance of this isn't really that much. But the fact that we can demonstrate it illustrates that I'm very serious about provenance and not willing to cut corners on where the distro content originated from. As a side note - we could have very well used Debian or Arch to bootstrap. In ways it would have made some things easier due to the FHS layout I've chosen, but that's mostly irrelevant now.

More later. Much more.

--sofar (aka Auke)