The Ur Project

Documenting the craft of stacking things so they remain stacked. For a very long time.

Out-of-tree kernel modules

Sometimes a driver you need isn't in the kernel. Or the vendor has a newer version than the one the kernel ships. That's what DKMS is for: it builds the module from source against your kernel, installs it next to the kernel's own modules, and rebuilds it every time a new kernel arrives.

This is best effort. Sandcastle ships the DKMS tool, but no module sources. Which module you pick is up to you, and if a kernel update breaks its build, that's something to take up with the module's upstream. Nothing in the system depends on a module you build this way.

Here are the topics covered in this guide:

Before you start

First, install dkms. It pulls in the compiler and the rest of the toolchain for you:

$ sudo pkg install dkms

DKMS also needs the build tree of your kernel. That lives in the kernel package's dev subpackage. It isn't installed along with dkms, because dkms can't know which kernel you run. If it's missing, dkms will tell you, and also what to install:

Error! Your kernel headers for kernel 7.2.13-3-stable cannot be found at /usr/lib/modules/7.2.13-3-stable/build or /usr/src/linux-7.2.13-3-stable.
Install them with: sudo pkg install linux-kernel-stable/dev
Or use the --kernelsourcedir option to tell DKMS where they are located.

So for the stable kernel, you'd do:

$ sudo pkg install linux-kernel-stable/dev

Every dkms command that builds or installs a module needs to run as root.

Register the module

Module sources go under /usr/src/<name>-<version>, with a dkms.conf at the top that tells DKMS how to build it. Many vendors ship one. Intel's out-of-tree ethernet drivers do, so we'll use ixgbe as the example:

$ curl -LO https://github.com/intel/ethernet-linux-ixgbe/releases/download/v6.5.9/ixgbe-6.5.9.tar.gz
$ sudo tar -C /usr/src -xzf ixgbe-6.5.9.tar.gz
$ sudo dkms add ixgbe/6.5.9

dkms add only registers the module. Nothing gets built yet. DKMS keeps a record of every module you've added under /var/lib/dkms, and it rebuilds from that record later on.

Build and install it

autoinstall builds and installs every registered module that isn't installed yet. By default it does this for the running kernel:

$ sudo dkms autoinstall
Autoinstall of module ixgbe/6.5.9 for kernel 7.2.13-3-stable (x86_64)
Building module(s)...... done.
Found pre-existing /usr/lib/modules/7.2.13-3-stable/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko, archiving for uninstallation
Installing /usr/lib/modules/7.2.13-3-stable/updates/dkms/ixgbe.ko
Running the post_install script:
Running depmod... done.

Autoinstall on 7.2.13-3-stable succeeded for module(s) ixgbe.
$ dkms status
ixgbe/6.5.9, 7.2.13-3-stable, x86_64: installed (Original modules exist)

The kernel already has its own ixgbe. DKMS sets that one aside, and the new one under updates/dkms takes over. That's what "Original modules exist" means: DKMS can put the kernel's version back when you remove yours. If you'd rather not install everything at once, dkms install ixgbe/6.5.9 does the same for just that one module.

You'll see two messages that look alarming, but they're harmless. Intel's dkms.conf uses a setting that DKMS considers deprecated, so it prints Deprecated feature: CLEAN. And DKMS says Running in chroot, modules won't be signed. DKMS builds in a private view of the filesystem (see How it works), and its chroot check mistakes that for a chroot. Sandcastle's kernel doesn't check module signatures, so there's nothing to sign anyway.

Load it

$ sudo modprobe ixgbe
$ lsmod | grep ixgbe
ixgbe                 573440  0

The kernel will note that something out-of-tree was loaded:

$ sudo dmesg | grep ixgbe
[  122.033728] ixgbe: loading out-of-tree module taints kernel.

That taint is a flag that shows up in kernel crash reports, so that anyone reading one knows that code from outside the kernel was running.

New kernels

When a kernel update arrives, you don't need to do anything. Installing a new kernel runs boot-manager update, and once that has set up the kernel, DKMS builds every registered module for it. And for every other kernel that boot-manager keeps, too. The log of that ends up in the journal:

$ journalctl -u dkms-kept-kernels

If a build fails there, it doesn't hold anything up. The new kernel is installed and bootable either way - it just doesn't have that module yet. DKMS also checks at boot, and builds any registered module that's still missing for the kernel you booted.

Each kernel's build tree comes from its own dev subpackage. So keep the kernel's dev installed for as long as you want to build modules. Old kernels don't lose their build tree when an update replaces them: boot-manager holds on to it along with the kernel.

A module without a dkms.conf

Not every vendor ships a dkms.conf. AMD's sfc ethernet driver, for instance, lives in their Onload tree, under src/driver/linux_net. Copy that directory to /usr/src/sfc-<version>, and write the dkms.conf yourself:

PACKAGE_NAME="sfc"
PACKAGE_VERSION="6.5.0.1000"
MAKE[0]="'make' KDIR=${kernel_source_dir} KVER=${kernelver} modules"
BUILT_MODULE_NAME[0]="sfc"
BUILT_MODULE_LOCATION[0]="drivers/net/ethernet/sfc"
DEST_MODULE_LOCATION[0]="/updates/dkms"
AUTOINSTALL="yes"

MAKE tells DKMS how the module builds, and the module's own README usually explains that. DKMS fills in ${kernel_source_dir} and ${kernelver} for each kernel it builds for. The quotes around 'make' stop DKMS from adding its own KERNELRELEASE to the command, because some module makefiles mistake that for being run from inside the kernel build. AUTOINSTALL="yes" is what makes DKMS rebuild it for new kernels. The dkms(8) manual has all the other options.

Then add, build and install it just like any other module:

$ sudo dkms add sfc/6.5.0.1000
$ sudo dkms autoinstall

Removing a module

$ sudo dkms remove ixgbe/6.5.9 --all

This uninstalls it from every kernel, puts back the kernel's own module where there was one, and drops it from DKMS's record. The source in /usr/src is left alone, so you'll have to delete that yourself.

How it works

You only need to read this if something goes wrong.

On Sandcastle, a kernel isn't a tree of files. It's a single image, /usr/lib/kernels/<kver>.erofs. boot-manager copies it into /var/lib/boot-manager/kernels/<kver>/ and mounts it read-only. On top of the image's module tree it stacks a writable layer, upper/, and that combination is what you see at /lib/modules/<kver>. A module that DKMS installs lands in upper/. The kernel image itself is never modified, and when boot-manager drops an old kernel, that kernel's modules go away with it.

The build tree is an image too: <kver>.devel.erofs. The dev subpackage installs it, and boot-manager keeps it next to the kernel. Normally only the running kernel shows up at /lib/modules/<kver>, so DKMS builds in a private view of the filesystem. In that view, every kernel boot-manager keeps is at /lib/modules/<kver>, and each kernel's build tree is at /lib/modules/<kver>/build. That view disappears the moment DKMS exits, so nothing it mounted is left behind. The FILES section of boot-manager(8) lists where all of these live.