The Ur Project

Documenting the craft of stacking things so they remain stacked. For a very long time.

On Lua

The codebase of this project is all rust. The package manager, dispatch, designer all share a large amount of code and when I started to build it out I knew I didn't want to build separate projects that had dependencies on out-of-tree projects that would quickly go out of sync, and be a nightmare to keep working and testing as the code was getting built out. So everything is in one large project.

A second realization I had up front is that I really wanted to kill any sort of "we can't solve this neatly now, let's just shell out here" as a fundamental solution. It's your rpm %post_install sort of patch work solution to "I don't know what's going to be needed, plan for everything" and the result is pure chaos and an unmaintainable collection of derelict scriptlets that do things in a chaotic way.

Not to mention that it's a fundamental risk. Post installation "massaging" of your system is inherently a bad thing. All deterministic shapes are just thrown out the window with it. And let's not even mention the security risk that comes with it, because you're now inventing a whole phase where nothing is off limits and everything goes.

Stateless was a good concept

Possibly it was a really bad name we picked for it. I don't even remember how we came up with the name, but the term is flawed as a choice for what it achieved: Zero configuration operation. This is a whole lot better than "stateless" but it's a bit of a mouthful. I do definitely want to start bringing this ZCO principle (anyone remember zeroconf? Maybe we need yet another name for this concept). Anyway, the concept stands: Things should work out of the box, not because there are configuration files in place making it work, but because there are not. The baked in defaults are the only thing needed for the thing to work.

But even I designed for an escape

So upfront I wasn't convinced I could do without the equivalent of the %post_install macro feature. How little faith I had. So I'm a big fan of Lua myself. If you've followed some of the OSS work I've done you'll know I've worked on a https://luanti.org based game called Inside The Box (https://luanti.foo-projects.org/) for over 10 years now. It's a puzzle solving, building and sharing server inspired by games like Portal, Mario Maker etc. And it's fully built in Lua.

Lua is a great programming language for embedding. It allows one to create a sandbox where execution is handled by the lua interpreter (which is very, very small) and the exposed system interfaces are purposefully limited by default - there is no real standard lua library per se. By default the lua environment is absurdly minimal. Perfect for locked down and restricted environments.

So the build system has this actual mechanism (not hooked up, but scaffolded) to do on-system post-install stuff. And then.... I never needed it. Almost 2000 packages later and not a single one actually needed any of it. Everything that did need some special first-time deployment thing turned into a system unit, or a tmpfiles thing, or just something else that's consistent and either idempotent or deterministic.

So, remove it?

Well no, there's actually one place I found that Lua turned out to be a decent solution. Surprisingly, it has nothing to do with building or installing packages!

The packages that flow into dispatch, they come through a component called designer which is meant as the ingest gate: it's the place where submissions get proofed: Do they build? What licenses are there? Any open CVEs? All that sort of thing is designer's territory.

Now, humans can give things to designer. Just edit a recipe in the web UI, and essentially you copy it into a designer space where you can safely experiment, build, reformat, and have it "run the gauntlet" of acceptance criteria.

And AI can give things to designer. Through the MCP protocol, any AI can prepare and submit recipes that designer will handle and check, before the person in charge gets to review them.

But there's also just "bots" - non-LLM programs that can stuff things into designer. Most importantly, it's upstream-watcher. And upstream watcher has a daunting task: It is supposed to monitor and submit version updates to packages that it monitors.

But I've run into the hairy mess of "watching HTTP directory index dirs" and having to make sense of "Is Odd Stable Or Unstable" and "a .90 release is pre-release" sort of tribal knowledge that varies per package. And that sort of stuff gets old really fast.

Now, I could opt into embedding all this information in yet-another-drop-in file that's kept in the registry. But I've decided that I do not wish for the registry to contain all this "fluff" data that is transient. There's not going to be any profiling data in the registry. There won't be test results in the registry. There won't be any sort of that stuff. It has to live outside of the registry, keeping the registry relatively clean.

To fix, the upstream-watcher now records a lua statement per package, and it is evaluated for every found potential update on the HTTP servers of the project. It compares the version number after parsing and the lua statement tells upstream-watcher to either ignore it (consider it unwanted because it's out of track) or submit as a version update.

There's only one C API function

Right now the Version track statement is a simple lua function that is called with v which is a table containing {major, minor, patch}. The return value of the statement must be a boolean. true means the update version is usable. false means the update version is to be ignored (out of track).

Here's an example one that is in use today:

return v.minor ~= nil and v.major <= pkgversion("gsettings-desktop-schemas").major

This lua snippet is in use in all of the GNOME apps. It signals the upstream-watcher that it should only consider updating GNOME apps with this statement if they have the same major version number as gsettings-desktop-schemas. This allows me to completely ignore GNOME-51 until I update gsettings-desktop-schemas from 48 to 51, after which it will automatically see, and file, version updates for every relevant GNOME application that has an update.

End Detour

Sometimes you have all the right intentions and end up building something that wasn't needed for its intended purpose. But the simplicity and restricted nature of a Lua sandbox makes it just so easy to use. The bottom line is that by and large we aren't relying on "shelling out" and opening a can of worms - we're letting a small and restricted scripting environment sort out a limited scope problem for us. The CVE watcher uses this feature as well for certain parts of the CVE/NVD unmapping problems. That one needs a lot more work - probably for a long time.

--sofar (aka Auke)