Architecture - The recipe format - part 4

1 tablespoon of salt. Stir.

Things are speeding up

We're starting to approach the skeleton of everything. It all boils down to content and how it transforms into output. The build system applies the transformations. These transformations are predictable. So let's not make it more complicated than it has to be.

Metric

The unit for our input content is the recipe. It describes what the build system consumes to produce output. Think of it as a high-level declarative thing. Mostly devoid of build instructions and compilation tweaks. And it's in TOML.

I've worked plenty with just about every way of defining structure in human readable text. And plenty of formats are super good for plenty of things, but when we're talking about humans writing any of them, they largely suck at it. What makes it worse is if there is some language subtlety that causes the program to interpret things differently than the human intended. So everyone should understand why YAML isn't a good choice for a declarative language.

JSON is a little too mechanical. I don't really like unordered lists and non-deterministic array order. Good for machines writing and machines reading. Not that good for recipes.

TOML hits the mark even if it isn't that elegant. It doesn't need to be, and there's tools that can help humans write TOML better. It's becoming really prevalent these days. So The Ur Project is TOML everywhere for simplicity.

The main dish

There's going to end up having to be some sidecars. It's almost entirely unavoidable that users want to modify some package build sequence and throw in a patch file, or some asset, and you just can't expect everything to be hosted in a git repo. I mean, we can try, but convenience is a thing. So recipes can be accompanied by [patch] sections and obviously they need [source] parts in the TOML. But everything starts with [recipe] and every [recipe] section contains exactly the same thing for every recipe:

[recipe]
name = "flagger"
version = "0.1.1"
release = 4
description = "Journal-tailing IP banner: scores log matches and hands repeat offenders to a firewall set"
homepage = "https://code.foo-projects.org/sofar/flagger"
license_hint = "GPL-3.0-or-later"
summary = """
flagger attaches to the systemd journal, filters it to the services its
rules cover, and matches each message against a set of regular
expressions. A match extracts the source address and adds a score to that
address's total; reaching the threshold puts the address in an nftables
set and its packets are dropped until the entry expires. Repeat offenders
earn progressively longer bans. Successor to tallow.
"""

The recipe identity for flagger is fairly bland

This is just the start, obviously. It needs to have an upstream and a record of its original content (checksum, signature).

[[source]]
name = "main"
url = "https://code.foo-projects.org/sofar/flagger/archive/0.1.1.tar.gz"
sha256 = "4150b9eec11c3e2e3660d2493554b591695ea51b43c0daa56321a7a2f106af01"
unpack_to = "."

A simple source URL file definition with checksum

Here's a typical patch section example:

[[patch]]
file = "0001-add-browser-build-driver.patch"
strip = 1
stage = "pre-probe"

[[patch]]
file = "0002-debrand-to-browser.patch"
strip = 1
stage = "pre-probe"

Some patches in the debranded firefox-based "browser" build

As you can see, there's an array of patch here, with multiple (ordered) entries. TOML keeps this somewhat simple - unlike rpm's spec format that makes you question your sanity every time a patch set needs rebasing.

Dependency declaration

There's a floor concept I've expressed in a few other places already. It is a way to express a bunch of dependencies that are implicit to make it easier to assemble buildroots, but also just prune all the required stuff from all dependency trees. But it doesn't contain everything - and more importantly, things need to be pretty exact to make sure you're not hauling in too much and thus turning on features one might not want in builds.

[build_deps]
needed = [
    "zlib",
]
host_only = []

boost only needs zlib headers, and no build machinery

[probe_args]
cmake = [
    "-DCMAKE_DOC_DIR=share/doc/cmake",
    "-DCMAKE_USE_SYSTEM_LIBRARY_LIBRHASH=ON",
    "-DCMAKE_USE_SYSTEM_LIBRARY_LIBUV=ON",
]

A few extra cmake flags needed to build .. cmake

The autobuilder probes can be influenced by passing additional flags to the program that the probe will call during the build or configure phase.

Is this clear where this is going?

Okay, good, so far we've seen how most recipe parts look: boring, declarative, and devoid of scripting. There's actually no scripting here for the most part. The one exception so far is just some basic "install this file here or there" sort of install: section stuff.

But by and large I'm just patching in a Makefile where things go wrong, or sometimes even a full meson.build. I dislike external scripts - they have no context and the source code has all the context - it's more reliable to just lean on that.

I cna't tpye

Good news, I built a linter into the dispatch-web interface where you can edit and submit modifications to recipes, and even start building new recipes, all right from the browser.

Recipe editor view: the format button does a TOML fmt

There's a built-in web-based recipe editor with live linter, it also can format the TOML according to its ruleset. There's a recipe glossary on the right hand of the edit pane that will show you valid TOML options and what kind of values they take.

There's also a "new recipe template" tool that mostly holds your hand with manually created recipes. It takes the tediousness out of entering new recipes and typing them up from scratch.

The new recipe tool - makes the work a little lighter

These are valuable tools that create speed and flexibility when you need it. Things like forcing a rebuild of a package or making a minor adjustment shouldn't be difficult. And even with LLMs doing good work, there's just nothing faster than a quick mind to fix a package issue.

Or just use $EDITOR

All of this can be done from a terminal - the place where recipes live can be a managed git tree, but it doesn't need to be. You can just have dispatch use a plain directory structure without version tracking. And you can submit your recipe changes through the CLI tooling if you prefer.

That's it, really

(There are quite a few knobs and tweaks I haven't shown for brevity's sake).

Recipes are almost entirely declarative. No scripting language, no substitution, if there is a need for that it should belong in the source build code, or in the autobuilder or probe components. That is where almost all the build magic happens already. We don't need to get overly creative and invent another macro language. I've got 1300 OSS projects' recipes written up and no macro magic was needed. As long as that holds, it's going to be smooth sailing. I'm not saying something might stump me later either, but I think we'll be alright for a bit.

--sofar (aka Auke)