All that matters now

We're in an interesting phase of the project. While I'm building hundreds of packages and dozens of images a day, we're not ready yet for a release. There's still serious foundational bits that need improvement.

The build system itself is the only thing that matters right now

Listen, this may sound like a weird thing, but, the actual distribution content is almost entirely irrelevant at this point. But let me explain this in detail so it makes sense.

Every bug is a schema rev and backwards incompatible

For instance, today I worked on an issue where the kernel image as installed didn't upgrade from one release to the next. The reason for it was that I had written the kernel images to be written as N-V, and not N-V-R. The problem here is that when R increases, the boot-manager code didn't see it, because the installed kernel artifact only had a key on N-V. So, a patched kernel with a critical CVE patch would go unnoticed by the boot-manager, and you'd be left vulnerable. Unacceptable.

But the fix for this issue is that the build code must insert the -release number into the embedded KVER symbol. And that embedding required that the kernel build itself puts the release number somewhere where the kernel Makefile code can reach it. And no such thing existed. So the right solution here is to do a setenv(). But this breaks if the builder isn't doing this, and it breaks reproducible builds with older pkg versions, and thus this is an incompatible API break.

This is just one of the minor architectural issues I've worked through in the past weeks. More will come into light. We'll get better at spotting them and I'm already making sure that I'm leaving systems in place to properly do all these schema-incompatible things correctly everywhere, because it matters.

Buildroot cycling is a mess

Every week or two or so I completely rev the floor of the distribution by rebuilding the recipes from the base registry. These are what makes the distribution work - it contains all the basic bits to start anew. And it also needs to move forward regularly because it's the fallback for everything.

The current rebuild of each new floor is trial-and-error style. It's highly inefficient, but it's successful. But only after a painful wave of build failures. These failures trigger a backoff mechanism - otherwise failures would just retry indefinitely. You can see where this is going - it's not pretty and it's a little sloppy.

So one of the things I'm working on today before going to the climbing gym is to overhaul the strategy that the convergence uses, and come up with a thoughtful approach that reduces the shape of the problem, so hopefully these buildroot waves aren't so tedious. I'd love to do them more often, but at the current rate it takes me about 2 days to do all 5 waves needed to stabilize them and get them past the promotion gate.

The repo server is ...

It's actually running. This is great news, because soon, I'll be able to build images with the correct URLs baked in, instead of having to use local lan DNS names that don't resolve properly in testing. Once they are populated, all artifacts can then point to the repo server URLs for testing and updates.

The repo server syncing was another problem I had to sort out. I'm unfortunately limited by my ISP's physical limitations to send network packets through the atmosphere, so my upload bandwidth is marginal. But even more so, I find out the hard way that my ISP limits individual rsync streams to 2-mbit/sec or so, and that isn't cool at all. At 2mbit it would take me days to upload a new buildroot repository. The actual complete Sandcastle repo set would take weeks.

A little trial and error however shows me that I can just run many parallel rsync streams. It seems that gets me about 20-25mbit or so. And that will be good enough. It means I can push updates quickly enough that it doesn't matter, and will not have to procure build infrastructure in the cloud, because that would probably cost me thousands of pecunias. Oof.

Full steam ahead then.

The obvious priority isn't to look at distribution content, but the build system itself, and make sure we can promise that everything works smoothly and according to my (high) expectations. Once it runs smoothly, I will release everything, and then we can go to town on distribution content, compiler optimizations, stateless implementation, immutable release images, and all that good stuff. But for now those big chunks of content work will need to wait a little longer.

--sofar (aka Auke)