The Ur Project

Documenting the craft of stacking things so they remain stacked. For a very long time.

Vulnerable

It's been quite a week. If you don't know what I'm talking about, you probably don't read much social media, or many tech posts.

We're all vulnerable

At any point in time, all of us are exposed to the possibility that some hacker may gain access to something we don't want them to have. This is the reality of today's technology. It isn't going to go away any time soon, because the reality of software is that it is very, very difficult to keep up with the work that is needed to avoid all this risk.

An absolutely massive amount of risk just stems from the fact that we have been writing insecure software, and never bothered to come back later and tighten up security around it. I'm saying we, because there are for sure pieces of software out there that I personally wrote, or that teams I was part of had a hand in. The cost of technical debt is never paid up front.

For The Ur Project, I'm leaning heavily on my experience as someone who worked hard on keeping users secure, and on mitigating risk. We did so in many different ways in the Clear Linux OS project, as any distribution should. None of those lessons are outdated. On the contrary, they're even more important today than they were 10 years ago.

If someone tells you there's a security issue, believe them

This is the ground rule for any OSS project. This is why there's a Security link at the top of every page. This is why I'm telling everyone that they can reach me when needed.

But that's just part of the whole process. Staying secure means that you adopt processes and strategies to reduce risk, and that you inform users of their risk. So today's post gives a little more information about this.

The process of keeping you, the user, secure

This leans heavily on several things we do in The Ur Project:

These let any user check our work instead of blindly trusting it. For this reason I post regularly about packages that failed to build reproducibly in our 24/7 reproducer tests. You can run these yourself and check that what we said went into the build of a package actually did.

Full validation of a package's content and its trust chain can be done with pkg verify, and you can get the full SBOMs as well, with pkg sbom. Check out pkg provenance for more details.

Our users rely on the tools they have (pkg) to make sure that they can avoid fake updates from a man-in-the-middle (MITM) attack, from someone who stole our update server, or from a DNS takeover. The Update Framework (TUF) is the mechanism that lets clients distinguish between valid and fake updates, even from a mirror or cache. It prevents you from installing things that aren't signed, or that would roll your system back to an outdated and insecure version. TUF and signing goes into the details.

The repo server is just a host running nginx that serves the repo data. Even if that server were broken into, the keys to sign updates aren't on it. You should still make sure that the images you download match the ones we publish on the main website, and the signatures guide shows how. And just to make things harder on any attacker: yes, the website where those checksums are is a different host.

The build server isn't directly on the internet. Even on that server, the parts that check for updates and download sources are heavily separated out. The code that creates submissions is what performs the download from a public source location, and that code can't touch what gets merged or the final repos. Even the test runners perform testing in isolation and the results they feed back are treated as untrusted data.

The repositories on the build host are only writable by the builder's own user ID and by the job that syncs them to the repo server. There is no passwordless sudo access. Any token used for accepting changes into the repos isn't stored on disk or on the same machine.

There are no automated merges. Coding agents can only propose. Interfaces exposed to submitters don't have the APIs needed to approve. That job belongs to a human (me), and Governance says it stays that way. I review everything. For this reason I built a very rich review panel that shows me exactly what's up in every change. It shows everything I need, even down to per-source changes inside the upstream tarballs.

All the keys that the build server uses are temporary keys. They expire. The only keys that don't are the root keys, one per repository. They are not present on the build server. They're not even online on any host. They are physically split across 2 locations, and only occasionally used to rotate the clerk keys that the build server uses for a few months at a time.

This ensures that we don't accidentally leak a root key by leaving it on a vulnerable machine, or somewhere else that isn't safe. It feels a little over the top, but remember: the root key signs every key we've used since we started, and rotates the clerk keys. Everything stops at it. Its fingerprints are on the Security page. For now the signed timestamp.json of each repo expires every 4 days, so nobody can freeze you on a stale copy of the repo for longer than that, and the clerk keys rotate every 6 months or so. We might shorten both in the future to further improve security for users.

The build host performs all the build functions, which means it downloads and compiles potentially hostile code. Therefore, all the compiles run in a fully isolated sandbox. No network. No downloading anything while building. The sandbox leverages several layers to protect against a possible sandbox escape: seccomp, Landlock, and several namespaces on top of that (PID, IPC, mount, user, UTS, network). Sandboxes run with their own tmpfs root, with no access to resources outside of it. Capabilities are cut down. The environment is cleared. All this is to prevent software from secretly pulling in something evil that could ultimately make it to your system in a signed update.

Strategies we use

There are a lot of things we do that aren't obviously related to security and keeping users safe, but ultimately they are. Many of the vulnerabilities that LLMs find these days are nothing more than (sometimes silly) little bugs. Code that behaves a little weird in uncommon use cases, for odd reasons. Things not left tidied up.

I've been testing our machinery, and because of how our build system is structured we're capable of turning around a major, impactful change extremely quickly. For instance, a major kernel update (like the one we had this week) was already out and available in our repositories before the kernel published its vulnerability disclosure. A kernel build takes about 40 minutes. Publishing the repos takes 3 minutes. Review is the only thing that remains. Everything else is automated.

Fast has a flip side, of course. A fix that goes out in an hour can break something in an hour, too. So if an update bites you, pkg rollback takes you back to exactly what you had before, and pins it there so the next upgrade doesn't undo your undo. And no, that doesn't fight with TUF's rollback protection: TUF stops someone else from moving you back to an old version. This is you choosing to, on your own machine, and it's still all signed packages that you have trusted before.

For each package we build, we run the package's unit tests. That's already a gap, because something like 30% of packages don't even have measurable or usable unit tests. A big gap is right there.

For each package change we rebuild images, and then test the images, sometimes in 3-4 different image formats.

For each build system or package manager change we test, and try to keep code coverage in tests high (90+%). It might not feel like this helps, but keeping it high will, over time, help spot negative effects quicker.

There are several systems on my network continuously doing image testing. These runners pull images that are created from proposed changes.

There's really not a good reason to run clamav on open source software code. But in general it isn't a bad idea to look for signs of malware in software. These could be things that are obviously bad, like code that attempts to use obfuscation, or just weird stuff that would pop out at anyone who looks at it. This is still something I'm trying to build out and experiment with, but we already look at the results of these scans before we accept packages into the build system, just like licenses must be scanned before we accept a recipe change.

This is the one I'm spending a lot of time on this week. Arguably it was the weakest link we had. Not because it was bad, but because it had way too many false positives due to the way it was implemented. The goal is to allow any user to see which of the software they have installed is known to be vulnerable, but this is actually not very easy. We'll continue to hammer away at this until we get to a point where we can show users how this looks. In the meantime, CVEs are being watched like a hawk.

This is upcoming. Obviously we could just throw tokens at it like every security researcher is doing, and we honestly hope that most OSS projects are doing something like this anyway. But because we all know there's going to be that five-year-old, half-baked, odd release tarball for that one base Linux OS component out there, it has to be redone. By us. We can't avoid it.

We've disabled old crypto entirely. We're disabling undesirable and unwanted startup services. Ports shouldn't be open by default, for many if not all services. Daemons shouldn't start by default or get pulled in by an accidental dependency. These stances are wide and have a long tail. That does mean that FreeRDP won't be able to connect to old NT4 systems, though.

The same goes for the binaries themselves. We launched with _FORTIFY_SOURCE=3 and a few basic hardening flags, and I already admitted that's the bare minimum. So we're pushing it harder: PIE, IBT/SHSTK, and stack-protector/clash. Every one of those turns a whole class of "silly little bug" from an exploit into a crash. A crash is a bug report. An exploit is a headline.

I don't actually mean "be responsive, publish a fix". What I mean is that some disclosures have a very long tail. For instance, the eMudhra emSign certificates that were revoked? It means making sure all TLS chains in the distribution are updated, as well as updating the builder itself, to make sure that the code that fetches software properly rejects bad certificates.

Fortunately, we didn't have PHP to patch, simply because we don't have it packaged. And honestly, I'd love to keep it that way for a while. We did have to update the kernel this week for a vulnerability. But there has been a real swell of disclosures. And it's still hurricane season for a few more years, I bet. The storm isn't done. It might not even be at its peak yet.

What's next?

I'm here tilting at windmills. Most of this post will end up as parts of the Security page, to explain our processes.

Reporting an issue to us is easy. There's a security.txt so tools and researchers can find the contact without hunting for it. The other direction isn't there yet: there's no advisory feed that tells you what we fixed, and when. For now that's these posts and the commit log. That's not good enough, and it's on the list.

At the same time I'm thinking ahead about better practices, and about streamlining the process for people who use the build system (even if that is just me), because all of this is served well by a build system that helps you deal with these issues. A build system that avoids dealing with these problems is everyone's enemy.

Stay safe :)

--sofar (aka Auke)