Everything must be tested.

I don't want to release untested code. Yet I have, and, it will continue to happen unless I expand the test framework to cover everything.

Did you test this?

No(*). Please read the footnote.

Footnote

Ah, you made it this far into today's blog post. Let me explain what's going on so you understand.

Testing is difficult. The sheer problem space of testing even a simple piece of software is complex enough to give mathematicians funding to obtain a PhD, a life-long career, and then more. Testing is fundamentally hard. And the alternatives are not attractive, either.

Many software packages don't even have testing code. Or they have a test framework that requires 20x the install base to even start the test. Or they require an active internet connection. Or they output their own test results in a completely nonstandard way that can't be easily collected by QA scripts. Or yet again some other thing breaks how this is tested that just won't work in a distro setting with hermetic builds.

Now combine 2000+ of these things together. Yep, it's absolutely a drama. I think the current packages have about (at best) 30-50% make check testing working. There might be more tests running, but we can't interpret the results. Or the tests take longer than 15 minutes and so it's aborted by default. I don't think I can allocate 15 minutes per package for 2000+ packages. That's 500+ hours, almost 21 days of testing, sequentially. Even at 6 parallel builds (which is what I run right now), it would be three and a half days of just tests.

So this isn't working for us. I can't even gate on failing tests right now. I can just hope that things work and maybe, maybe for some packages I can rely on these test results.

The failure, the test, and the gate

This is how you normally start. First you figure out that "huh, something is broken. perhaps I should make a test that detects that." So you write a test. And then you forget about the test. And then it breaks again. Possibly worse, and in production, or at a customer.

So then you understand that the test must run before it goes to the customer. Obviously. So you make the test result a gate. And now the fun has gone and it will never come back again. Every test failure turns into a thankless hunt for the little thing that broke it. Often, it is just something tiny. Many times, it is just the test that broke, and not the actual end user thing.

Hence, the gate. Good testing means that failures are spotted before the customer gets it, or it is deployed in production.

The rolling release model

Here we are. 2000+ recipes, 20k+ output packages, and it all should be tested. preferably before it goes out to users. So we have to gate acceptance of changes on test results. How do we do this, smartly?

First thing first. We can automate running per-package make check and see if tests regressed. Good. But as I showed you, it barely gets maybe 40% of all content tested. So 60% of packages can change and zero tests are in place to prevent accepting/merging a breakage.

In comes image-based testing. It's not mandatory, but, dispatcher makes it so easy that it's almost automatic. When you open a closure to incorporate several new changes, you can have images generated of all the combined content in this staging area. And these images are automatically picked up by test runners. The test runners will perform the normal image based test sequence on these images, and POST the results back to the closure. There, the operator can see the test results and use them to gauge quality. At least of whatever image testing code exercises.

Not complete enough

This last part (image based testing) has been in place for 4+ weeks now, and it has been immensely valuable to gauge continuity. But it still irks me that we have so little per-package functional testing in place. And really, there isn't a good reason for that, other than time.

But, we live in the age where testing whether things work and writing it out in scripts isn't human work anymore. Even back in Clear Linux OS, a large part of our testing was nothing more than "run every binary with --help, --version and make sure it shows something" as well as "run ldconfig and make sure no library is missing any dependencies". As of today's age, we can expand that into asking your favorite LLM to write a simple test script that validates that mkfs.vfat creates something that looks like a vfat filesystem that can be mounted, and that sqlite3 can perform a bunch of basic SQL queries against a known test database.

Is this trustworthy? Maybe not. Is it signal enough to be worth the effort? I'm betting yes. Any test that succeeds more than three times and then fails on a change is worth looking at. Maybe the test needs updating, but, it offers better visibility into changes in the packages.

So that's what we're building out

I've posted before that I currently have 4 systems dedicated to doing 24/7 testing. This testing includes

• reproducibility audits. Packages are recompiled and compared against the original compiles. They must match byte-for-byte.

24/7 reproducibility testing audit's dashboard in dispatcher

• image testing. This is largely very high level quality testing to see if nothing major broke. If it doesn't properly boot, and look ok, it's a show-stopping bug

Per image testing results in dispatcher.

But that's all these runners were doing. All the per-package tests were done on the build server, right after the compile phase. One box, stuck testing autoconf for 15 minutes after it compiles, consuming a worker slot. Having per-package functional testing out of band frees up so much time while at the same time allowing multiple runners to do potentially different testing things. Like dependent package testing: If a library changes, it could test all consumers of that library, across several nodes, in parallel.

Btop running on a 2-core gemini lake with 4GB RAM - the smallest deployed test runner in my test farm. If it works on this tiny little box, it probably works everywhere.

Another problem I am having is that structurally my test framework put all the test content into the harness. And that's not workable. I also just had most of the test code hard written in rust. Which is probably a major sin of sorts.

The future architecture that I'm building out is that tests are stored in a central library, as Lua scripts. The lua scripting environment is a sandbox that is restricted and only has limited ways to interact with the DUT (Device Under Test) for security reasons. On the DUT, a simple probe executable runs whatever the lua sandbox requests it to do. If this crashes the probe, the lua sandbox is unaffected. If the probe sends back invalid data, it isn't run through unsafe shell code, and the rust harness throws it out.

The outcome is that I can deploy a central library to store test content and have it reach many test runners. The test runners still talk to dispatcher to look at package test opportunities and post results, and at the same time, the library can also host goldens and other payloads that the lua workloads can consider, like test blobs and injectable things that are too large to embed in Lua.

The amount of things we'll be able to test will vastly increase, and we'll be able to test way more things before they land in a user's update on their own machine. A solid gain for quality. It's almost all development time, too. From a review perspective, we'll have to weed out false positives for a bit. But with this LLMs can help to distinguish between the noise and do reproduction cases.

I'm close to ready to deploy this last harness/library architecture on my own test farm. When I'm happy it solves the problem I want it to solve, and in the way that it does (secure, flexible, easy to use), it will be posted as a new project under our Forgejo instance, so that everyone can easily deploy it.

--sofar (aka Auke)