The Vendor - Admin split

It's been well over 10 years since we applied something we called "stateless" to Clear Linux OS. But the ecosystem still hasn't picked it up. Unfortunately.

The Vendor - Admin split

Normal Linux users are generally entirely confined to their home directory. There is essentially no place outside of it that they can modify files. This makes it super easy for users to do backups and even a full OS re-install: just back up your ${HOME} and restore it after. Or even put /home on a separate drive or partition.

But for the Admin it's a little more complex. When an Admin performs a full OS upgrade, often system services will have issues with the older configuration file schemas/formats. Or they might just work, but leave the system in a terribly insecure or unstable situation. Or they might even do worse.

The root of the problem: endless proliferation of packages that require that some sort of undocumented, unschema'd and not upgradeable configuration files are put into an unorganized soup of config files in /etc.

A solution?

Yeah, and it's not that complex. At least not the concept of it. All there is to it is that we tell software:

1. To work out of the box without requiring any files or folders in /etc.
2. To have sane defaults that do not leave the user in trouble, or insecure.
3. To not mess with administrator-provided files in /etc and always use them.

Our own pkg already works this way: with no /etc/pkg.toml at all it runs on its defaults, as pkg.toml(5) says.

Easy enough. Why isn't this done in every software package?

Well, for one, a lot of software is older than this concept, or at least, older than e.g. systemd which put a significant amount of work into adopting this, at least in a few major Linux OS subsystems. But a lot just got missed. Or it was never done.

Are we sure we still should do this?

Absolutely. For one, it's not really a good idea to have 500+ configuration files in /etc when you only run 2-3 services and modified a single configuration file.

The upgrade issue is a much bigger problem. Most configuration formats are not schema-marked: there's no way to mechanically upgrade an older configuration to a new version even if a replacement guide exists. It's purely manual work. But the problem almost goes away entirely if services and daemons adopt a drop-in-style configuration format where stuff just works without any configuration file to begin with.

It's also much easier to do a factory reset style change. In a stateless sort of system the files in /etc are optional. Erase them and you get the defaults. This becomes a factor in a read-only /usr scenario. Atomic update style distributions don't want to maintain a complex safety net for upgrading service configuration files.

What is the risk?

The risk is that services and daemons break when upgrading from the old method to the new vendor-admin-split. Many packages need careful testing and patching to make sure things don't randomly stop working. So it's not as easy as patch and deploy. The good thing is, a lot of work was already done by me and the former Clear Linux OS team, and those patches are straightforward to adopt again.

Another good reason to do this now is that we have a solid base, but our package count isn't so large that this becomes a huge undertaking. We're under 2k packages in Sandcastle and very little of that is complex service/daemon terrain. This will make for easy and solid progress in the next weeks.

--sofar (aka Auke)