Documenting the craft of stacking things so they remain stacked. For a very long time.
Over 10 years after Clear Linux OS called it "stateless", most software still needs a soup of config files in /etc to run, and upgrades break them. Software should work without any files in /etc, with sane and secure defaults, and never touch what the admin put there. Why that matters, what it risks, and why Sandcastle is starting on it now.
A week after launch. Six new images fix the installer's initramfs failure on the ISO. GNOME 51 arrives with a batch of GNOME Circle apps, along with Plasma 6.8 and kernel 7.2.11. USB sticks no longer stall at boot on hwdb, and three of four reproducibility failures are cleared. pkg gains proxy support, and dispatch can roll back a bad release. The website got a thorough overhaul.
The first few days after launch: bug fixes, refreshed images, a new download page and image archive, and what comes next, from hardening and LTO/PGO to GNOME 51 and how to contribute.
In the past day we've posted everything you always wanted to make your own Linux Distribution - the builder source code, the package repositories, the registries and the images. The first distro that comes out of the build system is our rolling release, and it is called Sandcastle. Secured by TUF, reproducible builds always, no license surprises, and you could build your own today.
The image files are on the Download page. Go ahead and hop over there right away, and give it a try. I'd love to hear about your first experiences - if you can, hop over to the discord and tell us about how it went.
This git project contains the whole OS builder (named dispatcher), the package management utility (named pkg) and the composition/image build software (pkg-compose). This is the software that builds everything. It is a little too complex to explain fully how this all works, and that will come in the next few days. The easiest way to get started is to wait until the live USB images are available.
The compositions are published. The build system compiles sources, using recipes, into packages, but packages by themselves are just files in a specific format that are stored in a repository. The composition files describe how packages are combined and shaped into images and other things - a transformation layer. For now that means container, generic cloud, live USB, ISO and WSL images, plus a pins manifest output.
The base and common registries are published. These 2 registries are what contain all the recipes, and a recipe is the identity of how a package gets built and what it is. base is the registry of the floor, and it builds against base entirely by itself, which gives you fully self-sufficient bootstrapping. common is just everything else, for now. Both become useful once the images are published.
It's coming.
I'm calling the Docs and Guides good enough for the first release, with all the important guides fully written. The reproducibility issue I wrote about earlier this week turned out to be mostly fixed by rebuilding everything with the last builder code, and of 1731 unique packages tested only 3 do not reproduce and 3 have errors. Next up is one good crack at wrapping up the open issues from the final code review.
Tiny short list today just so you know I'm not disappearing but putting my time into things I said I would do before release. I've posted the Deploying the build system user guide, which ticks off another task I marked as critical for the release. And I'm doing some major rebuilding to make sure that our promise of reproducible builds is going to hold.
In the development of software, one will always accumulate technical debt. I started back in May, took about 6 weeks to get a fully booting OS, and a whopping 3+ months since then has for a very large part been nothing more than mopping up technical debt. Reproducible builds are one of the things that still need shoring up. There's no way I can make something "good" and "fast" and "cheap". I'm going for good.
In the next few days I'm going to rapidly wrap up the release. No private beta - that would just be me waiting for other people to do hardware specific testing, and that doesn't accelerate things. The new shortlist is image installation testing, one last re-review of the code changes, a bug tracker on forgejo, and the build system deployment guide. Posts are going to be a little short while I bring those to a close.
I've been spending my time writing out the documentation that needs to be in place on day #1. The minimum set of user manuals is an installation guide, how to use the package manager, and how to run the build system and make your own content. The documentation website is being built out now, and I'm focusing on adding at least one major user guide document every day.
The codebase is all rust, and I wanted to kill any sort of "we can't solve this neatly now, let's just shell out here" from the start. So I scaffolded a Lua sandbox as the escape hatch, the equivalent of the rpm post_install macro, and then never needed it - almost 2000 packages later, not a single one. Where Lua did turn out to be a decent solution is upstream-watcher, which evaluates a statement per package to decide whether a found version is on track or should be ignored.
Wrapping up the mechanical parts of the TUF implementation - physically securing the keys. The master keys are air gapped, split into two and stashed away sensibly, and the keys on the repository server now roll forward automatically, so if my build system goes offline for too long every client rejects the updates. That is what protects you against my server getting broken into.
Final code review is underway and user guides have started, plus a new item on the list - final publication. The dispatch deployment is getting built and blown away over and over to make the setup guide solid, and some of those steps will stay in a terminal because provisioning keys and tokens can't be a mouse click. The documentation site is still the thing I haven't gotten to.
Five months of 60-80 hour weeks doesn't buy perfect, so here's the shortlist of things that still suck - rust packaging where matching crate versions costs days while the builds take seconds, no cross or native non-x86 builds, no immutable distro because the choices aren't made yet, no dashboard for non-build failures, and nowhere near enough documentation.
Over 1500 packages means roughly 10k license files, and no, they are not all the same dozen licenses - some tarballs carry over 2000 license declarations. Checksums catch the easy copies, askalono narrows the field, and then someone has to make a call. So license-api hands that last pass to an LLM, which files an opinion the reviewer blesses or rejects.
Code review day, so no in-depth article - but the pace is high and issues are closing out at a good pace. Fun mention - Steam through Flatpak works fine on the Starlabs Tablet, with Stardew Valley, Slime Rancher, Bloons TD6 and an old Doom II all running.
The omissions answer the question better than the inclusions do, so start there - no atomic /usr yet, no native Steam, x86_64 only, and no ZFS or proprietary NVIDIA. What is there - GNOME, KDE and Xfce4, a de-branded firefox with uBlock Origin and DuckDuckGo, and plenty of console basics.
A distro is a random pile of packages until it isn't, and the difference is a goal plus something to enact it. So the first product gets a name - Sandcastle, the rolling release, carefully built and rebuilt whenever the tide comes in overnight. No LTS branch yet, and no promise of one until there is someone to help carry it.
Why the project runs on The Update Framework, and what it actually defends against - injected payloads, rollbacks to a known-vulnerable version, an update that is newer but still exploitable, and the day someone walks off with the signing keys. TUF assumes that last one happens, which is why keys roll constantly and a stale repo is one clients refuse.
Six of the nine release criteria are ticked off. The repo server is live on the internet - closed to browsing, because bots would flatten it - which unblocked images, updates and tests without a LAN mirror. What is left is documentation, user guides, and a new last item: a two to three week comb through the whole code base while breaking changes are still free.
A distribution lives or dies on its content, and content costs time - which is why the recipes here are written by LLMs and not by me anymore. Designer is the isolation layer that lets agents submit, iterate and retract without deciding anything, and every proposal still lands in front of a human with license, malware, CVE, test and build scans attached.
Paseto tokens instead of a full account system - minted with one command, pasted into dispatch-web, and they carry the role with them. Then the admin TUI that grew out of not wanting to remember the flags: seven tabs for the work that is too dangerous for a web button, including a sweep that reads every file and checks every checksum.
After 200-300 packages I could no longer remember which options I had enabled where. So dispatch-web now has full text search across every recipe and package - files, source URLs, provides and requires, patches and compile options - fed by each build as it publishes. Type a configure flag and see exactly which recipes carry it.
A tour of the four images coming out of the build system today - an 87MiB OCI container with pkg already in it, a qcow2 that wants a cidata drive before it will let you log in, a 352MiB ISO that still boots legacy BIOS, and a GNOME live USB whose installer takes an empty disk and nothing else. All of them go through the test runners.
Hundreds of packages and dozens of images a day, and the distribution content is still the part that doesn't matter yet. A kernel upgrade bug that turned into an incompatible API break, buildroot waves that take 2 days and five tries, and a repo server that is finally running - fed by enough parallel rsync streams to get past an ISP that caps each one at 2mbit.
The dispatcher builds images automatically, as soon as an image could be updated. What a composition is, a simple composition.toml for pkg compose, the composition list and test results in dispatch, and why immutable system images have to wait.
End of week 3, measured against the same four release criteria. A Live USB that boots and installs - on two machines so far, and it took manual work to get there - with WiFi and the touchpad working. A local package build demoed. Automated testing of the build system next. And the item that decides whether anyone else can use any of this: documentation.
A video, and not much else. Forty-nine seconds of a package built locally with nothing but the pkg tool - the source fetch, the recipe, a signing key, the build itself, then a bundle installed on the machine and run. The same code the build server uses, with no build system deployed anywhere.
A repository is a handful of content-addressed stores and a metadata tree. Why the checksum is the address, how that makes duplicate content free and unchanged subpackages a no-download update, and what the other four stores - licenses, provenance, recipes and sources - are there to prove.
What comes out of the build is split in two: a signed TOML manifest that describes every file, and a plain zstd tarball stored under its own checksum. Why that makes a rebuild that changed nothing a free update, why there are no timestamps, and how bundles carry it all across an air gap.
Every package is split into subpackages, and pkg installs the core ones by default. How pkg.toml can pull in man and doc pages automatically, why pkg list shows what you chose to install instead of everything that came along with it, and how sets nest and follow their changes during an upgrade.
The shortest one in the series, because a registry really is just a box of cards. A directory of recipes, a registry.toml that indexes the ones which count, and a sets directory holding the package groups that pkg tracks by intent when you install them.
A fixer upper should cost about what the job is worth, and so should changing a distribution. The tiers of modding The Ur Project, from a local package build that needs nothing but pkg and a few build tools, through derivative and overlay repositories, to a rebuild for a different ABI and a completely independent OS.
Two weeks public, and a walk back through the four release criteria to say what moved and what did not. Automatic image builds, a debranded browser, updates working from the Live USB, an admin TUI for the tasks that should stay out of the web UI, and the repo server that still has to be bought and stood up.
The recipe is the unit of input for the build system: one TOML file, declarative, with no scripting and no macro language. Why TOML beat YAML and JSON, what the recipe, source, patch and dependency sections actually look like, and the browser-based editor and linter that take the tedium out of writing them.
A survey at the end of the call, and why the answer here is no. Clear Linux shipped a genuinely privacy-preserving telemetry client and users hated it anyway. The Ur Project collects nothing: no appetite for it, no lawyers to bring in, and a good test pipeline catches more than crash reports ever will.
A pile of blocks only stands up if every block is checked before it is stacked. The gates content has to pass to get into the Ur Project: license scanning, malware scanning, the compile, reproducible builds, package tests, vulnerabilities, and the review page that puts all of it in front of a human who has to decide.
Koji made large package changes serial and painful. Dispatcher instead spins up throwaway repositories that resolve dependencies like real ones, so a hundred interdependent recipes can be staged, built and accepted as one group. These are closures.
What dispatcher is and what it does: a job scheduler that watches registries of recipes and turns them into repositories of packages. The graph view that made the topology workable, and the long bootstrap from a Fedora seed to three bit-identical buildroots in a row.
A week of being sick, starting a new job, and watching Phoronix and Reddit pick up the project. What readers saw that isn't there: this is an opinionated build system, not an opinionated distro, and a project where companies sit down at the table with the rest of us instead of across from us.
Why there is no epoch here. How name-version-release keeps every package in a content-addressed store unique, and how publishing one list of "current" pins turns upgrades, removals and downgrades into the same operation.
A hard status update instead of a conceptual one. Where the code will live, what still stands between "it works" and "it's ready for consumption", what the unknowns are on hardware, and what moving the build system onto its own product broke along the way.
Why this project is written in Rust. Some notes on what C asks of you, what Rust's tooling gives back in test quality and type safety, and where scripting languages do and don't fit at the lower levels of an OS.
Picking a license is the easy part. Naming the thing is the hard part. Why the project is named after a very old, very weathered stack of rocks in Iraq, and what that says about playing the long game.
A little bit on vocabulary - explaining recipes, registries, packages and repositories. What a Content Addressed Storage model buys us (airgapped updates, and deduped content). And why sharing content is our first class citizen.
What the project actually is - one Rust codebase that both builds the distro and consumes it, a tree of repositories rooted in a self-rebuilding build root, and a consumer side that is more than a package manager.
Nineteen years of packaging by hand, what finally got automated away, and where LLMs do and do not belong - not in the build system, not in your distro content, but in the boring parts around them.
The output isn't the point yet - the builder is. Why a serious distro has to make statements about licensing, CVEs and reproducibility, and why the build system comes first.
Nineteen years of building Linux distributions, over 25 years of time, four distros alone at Intel, and why I'm starting yet another one - and what I think we got wrong the last time.
