Architecture - dispatcher - part 2

September 7, 2026

I've made some comments before that I found it complex to do larger changes package wise in Clear Linux OS. Part of that is that koji made it elaborate. In koji, at the time, we had it setup so that we could push changes into it. Everything landed in a single repository. If you pushed something that required something else to be rebuilt, it'd take a script to detect it and also do the rebuild, and then so on and so forth. So if you pushed a big change like a new libcrypto.so that had a ton of dependents, let's just say nothing else got pushed for a while, because it would be busy rebuilding.

Now, you could still do scratch builds. So we ended up stashing scratch builds and then manually extracting the RPM's locally and hope it all worked (literally using rpm2cpio $FILE | cpio -idv over the rootfs. Because fast. swupd would clean it up later, if it landed. In any case, once you accumulate a bunch of scratch RPMS you still had to push them, one at a time, in the right order, into koji, and then watch it for breakages. And then fix them. Or the wrong order and hope it didn't get stuck.

Scratch repos

In dispatch, we fortunately can easily create many repos, and they can have long lives or be short lived. This by itself already solves the usability problem I mentioned. But, dispatch also creates essentially scratch repositories that are fully functional for dependency resolution on the fly for each pending package update, or new packages.

Dispatcher will treat these scratch things like they're real repositories, and build recipes sent to them as if they're another repository layer in the same way as the floor chain mentioned in the last article. You can see where this is going: these scratch things allow you to stage many interdependent changes all together in a single pass.

I've had a few of these now with about 100 recipes in them at the same time. This sounds like a lot but when you package rust-based software we easily use 200+ crates just for dispatch itself. So it adds up. But more on Rust packaging later. GNOME for instance was done in 5-6 of these scratch chunks to break up the tediously long dependency chain. Each of the chunks was 5 to 20 recipes. KDE/Plasma went in in half. It's much shallower actually.

Accept as a group, or, separately

Now, as these things are built, they pass certain gates. Only when everything looks good, the experiment becomes a proposal. proposed targets can be accepted. And acceptance is what it looks like - it gets merged in common or base. And then dispatcher builds the final version into Sandcastle-Base as needed.

Dispatcher can accept things from the scratch boards in batches. This allows things to land together instead of broken parts. Once a recipe gets promoted/accepted, it will take a little bit of time (or, a lot, sometimes!) before the output packages are available. Until that part is done, it can't provide anything, and stuff might break. So merging things as a group is a good thing.

Closure

I've named these scratch repos closures. They're a fundamental extension to help developers stage content changes into the proper repos. They are simple to understand and help you build complex changes before they land, and thus allow you to test them together. This builds confidence in that everything will work together before it's put into production. The obvious next thing to implement is image generation directly from closures. This isn't done yet, but it's really one of the highest value targets left to do.

Some recent closures showing in the dispatch closure view web SPA

I've made hundreds of these closures already. Some with very large amounts of changes. Being able to subdivide the work and process items in chunks has been hugely positive. It gives me the context and data I need to review the changes incoming.