Architecture - Content review - part 3

A pile of blocks - is what I've been paraphrasing a distribution as. That is essentially right. It's also a little bit of an understatement. A pile of randomly sized and oriented blocks could just blow away. If there's no cohesion between the blocks, it's like a star dune in a desert - it will literally blow away over time.

What makes a distribution good is when the criteria for "is this block ready to be stacked?" is strongly defined, and checked - for every block put in place. The things that cause a block to be rejected could be mechanized, but largely humans desire to eye-ball it and call it good. That's certainly a good way to do it if you are detail oriented and have a ton of experience, but it might fall apart if you're tired or you just need it to stop being spectacularly broken. And then it gets worse.

A bunch of gates

Any content that comes into the Ur Project build system must pass a bunch of gates. Some are fairly obvious gates. Some are not so strictly defined just yet, but all of them contribute to a decent review.

The License scan

Before the content even makes it to a trial compile, content must pass a license scan. For obvious reasons. For a community run OSS distro, we tend to largely allow pretty much anything because most of it is OSS licensed anyway, but there might be a "good, bad, and ugly" sort of thing hiding in things like media codecs, and people who took decent OSS projects and changed the license in a way that made a lot of people sad. So, there has to be a gate. The minimum gate is that the scan was run, and that the scan results were good enough. If the scan results were not good enough, a human has to go in and take a look.

The license scan and disposition screen

This dialog shows a package that wasn't properly recognized as HPND. The operator can compare the found license file against SPDX expressions and file it to surpass the gate issue. The resolve button then files the operator decision. The scanner relies on askalono to do the scanning and recognizing. There's no LLM involved here. It's just programmed pattern matching.

The Malware scan

It'd be terrible if the xz thing happened again. There are other examples of compromised upstream OSS projects where people have introduced malicious code, and nobody recognized this. This is the new reality for OSS projects. We have to figure out how to deal with this problem. In Clear Linux OS, we had to do virus scans. But virus scanning is a different attack vector - generally viruses don't come to your box through the package management system. They happen through file exchanges by e.g. email or USB stick.

This isn't going to be a good solution going forward, and it doesn't even protect against the xz type vectors. What does give some value of protection is something that gates at ingestion and throws red flags at things that look really bad. Then we'll just have to fight the false positives. I've done a survey for things we could potentially detect and I'm exploring more things - even things like "prompt injection" scans aren't off the table.

Samba sources compile differently in an RPM sandbox - could be evasion!

The compile itself

The submission needs to be compiled. That needs to be successful. If it doesn't compile, it doesn't meet our standards. Simple enough.

The reproducible build

All builds must be reproducible. This means that bit for bit, the manifest produced at the end of the build, matches exactly. This is a little more tricky than it sounds. The manifest of a package in The Ur Project contains the content checksum and shape of the content on disk - except for the inode timestamp data. It captures permissions, ownership, attributes, size, bits inside the file, all truthfully. But it doesn't capture the inode atime/mtime and ctime. Because those will always differ between builds. They're not even stored in the manifest for that reason.

There's a bunch of things that the build system has to get right to make this work. Most commonly, SOURCE_DATE_EPOCH needs to be set to the same value for both builds. So that value is seeded in a consistent way in all our builds. We also need to insert ar and ranlib flags to make those programs deterministic. A few other packages have other problems that must be addressed.

Tests - if you can.

We run all package unit tests. Well, um, yeah, if there are any, and some times we do and we can't automatically parse the test results. And yeah this is a total shit show really.

There's about 120 ways of running tests per 100 packages. Running package tests is a good idea. A lot of packages do a good job of making it easy to use and not do weird stuff like curl out to an internet host. Because that won't work in a proper build sandbox.

Package testing is in a terrible shape by default - many packages have their own custom test methodology and reporting format.

This is best effort. We do gate when there's a ton of package test failures all of a sudden. This is clearly WIP and needs improvement. Not everything is perfect yet. At least the mechanism to start upping the quality on this is there. Now we need to do a long haul of looking at every package that fails to return test results.

Vulnerabilities

Big one here. Not an easy one, but one that is absolutely needed - reporting at a minimum whether packages are known to be vulnerable to certain things. The most common form is CVE's. These are public information. We can query open CVE's against any package. It's something we just have to do. Even if we just relay the vulnerability data to our customers, we've already taken the uncertainty away.

How this all ties together

All of these gates are useful data. A reviewer has to decide whether despite any failures in any of these, or flags raised, is acceptable or not. If they don't have the data, they can't make an informed decision. The review page therefore carefully shows the reviewer the summary of each of these items and presents the findings and their detailed content. Then the decision is informed, and not random.

This screenshot shows the meat of the review content presented to the operator. It's nothing special, just sections with the critical bits at the top showing the findings of the various gates listed earlier, and links that will take you to the detailed findings. The bottom half expands further below and shows a quick overview of the package changes and allows a reviewer to gauge whether the content meets the expected results - e.g. the proper SONAME provides are present, the package links against the right libraries, provides the needed developer bits, or even specific subpackage changes that the operator is looking for.

For a simple version bump, there's some significant changes in this particular package, but it's likely due to the major version update of the nvme-cli package - since it takes it from 2.16 to 3.0.

There's more data at the bottom of that section. The full content of the files and the output content can be inspected. As well as a history of the changes and links to build logs. All these things help me to review hundreds of package changes efficiently, since they show me what matters, and not what doesn't.

--sofar (aka Auke)