Add Linux kernel smoke test #3447
Open
+2,316
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
linux.be0 configures with `make olddefconfig', which answers every new or
newly visible symbol with its default and never asks, so a symbol that
upstream renames, splits or re-parents turns off without a word and its
module is simply not built. The update from 6.12 to 6.18 cost several
modules that way, and nothing noticed until the hardware they drive was
used. `make oldconfig' would have asked, but it cannot be run unattended.
The script is meant to be run on a machine rebooted into the new kernel,
before the kernel is rolled out further, and looks at it the way the
regression shows up in practice:
* every device in /sys is matched against the running kernel's
modules.alias and against the reference kernel's, so a device that
had a driver and has none now is named, whatever the config says;
* the modules of the previous kernel are diffed against this one, with
a table of known upstream removals to keep the list short enough to
read;
* symbols the previous kernel had are split into the ones olddefconfig
turned off, which are ours to turn back on, and the ones upstream
deleted, which are not;
* a new symbol answered n that gates a feature still configured -- the
_LEGACY pattern -- is reported on its own;
* 113 config symbols the site depends on, each with the reason it is
listed, from autofs and NFSv3 to the BMC console and k10temp;
* the modules /etc/modprobe.d names have to exist;
* NFS, the automounter, cgroup v2, an overlayfs mount in a user
namespace and a veth pair in a network namespace are exercised;
* dmesg, failed units, sensors, EDAC, IPMI, cpufreq and the clocksource
are read for what a wrong kernel leaves behind.
--save-baseline writes what a machine looks like before the reboot and
--baseline compares after it, which is the only check that knows what the
old kernel really did rather than what it could have done. --kernel
checks an installed kernel against this machine's hardware without
booting it first.
Everything works as an ordinary user; what needs root says so and is
skipped. A run takes under two seconds. On a 6.12.98-492 machine it is
silent, and the kernel we are moving to reports, among others:
WARN CONFIG_BRIDGE_NF_EBTABLES_LEGACY is new in 6.18.51.mx64.498 and
not set, while CONFIG_BRIDGE_NF_EBTABLES is m: whatever it
gates is no longer built
which is this bug caught in the act: no ebt module is built since 6.12
although the config still says CONFIG_BRIDGE_NF_EBTABLES=m.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
Reading a kernel package tells us which modules were built; it does not
tell us whether they load, whether iptables still filters, or whether a
filesystem can be made and mounted. Until now that needed a machine
rebooted into the new kernel, which is the one thing nobody wants to do
before the kernel has been tested.
--vm boots the package in qemu instead, on any machine, without touching
the running kernel and without root. The guest is an initramfs built out
of this machine's own binaries -- bash, mount, ip, iptables, mke2fs and
the rest, with their libraries resolved by ldd and the xtables plugins --
so the test is the site's userspace against the new kernel rather than a
busybox that mariux does not even have and that has no iptables, no
mke2fs and only a stub ip. The modules come out of /lib/modules/<release>
with modules.dep parsed directly, and depmod -b rebuilds the indexes kmod
needs.
The guest then, as PID 1:
* loads the 26 modules the site depends on, from ip_tables to nfsv3;
* counts real ICMP packets through an ACCEPT rule, shows a DROP rule
dropping, puts a MASQUERADE rule in the nat table, matches
-m conntrack, and does the ICMPv6 equivalent with ip6tables;
* carries IPv4 and IPv6 over a veth pair into another network
namespace, puts a port in a bridge, brings up an 802.1Q interface
and reaches the gateway over the virtio NIC;
* makes and mounts ext4 and XFS on virtio disks, writes, remounts and
reads back, mounts a squashfs image, copies a file up through
overlayfs and does it again inside unshare --user --map-root-user,
as the rootless bee-file builds do, and checks POSIX ACLs and the
cgroup v2 controllers.
--vm-sweep additionally insmods every module in the tree -- 747 in
6.18.51.mx64.498 -- and, because a module that fails in a VM usually only
means the hardware is absent, boots the reference kernel the same way and
reports only what loaded before the update and does not load now.
A run takes about fourteen seconds with KVM. On 6.18.51.mx64.498 it is
quiet except for the ebtables regression, which it now catches as
something that breaks at runtime rather than as a config difference:
FAIL ebtable_filter is not in 6.18.51.mx64.498's tree at all; the
VM needs it for ebtables, which no module has been built for
since 6.12
SKIP netfilter: no ebtables in mariux, only the modules can be
tested
FAIL (1 failed, 7 warned, 1 skipped)
and the sweep agrees with the kernel it replaces:
OK sweep: 732 of 747 modules loaded, 15 did not, all of them in
6.12.100.mx64.493 too
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
--vm uses /dev/kvm when it can read and write it, and emulates otherwise.
The emulation path had never been taken, because every machine here has a
usable /dev/kvm, and there was no way to ask for it; a machine whose KVM
is broken rather than absent had no way around it either.
$ ./smoke-tests/linux-smoke-test --kernel 6.18.51.mx64.498 --vm \
--vm-no-kvm --only vm
WARN no usable /dev/kvm, falling back to emulation, which is slow
enough to need --vm-timeout
...
OK the VM ran the tests and powered itself off in 21 s
FAIL (1 failed, 1 warned, 1 skipped)
Everything the guest does passes under TCG as well; it takes 21 seconds
instead of 8, and the remaining failure is the ebtables one.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
Collaborator
|
we already have a |
On a machine with an NVMe disk, the storage check warns although the
disk is driven perfectly well:
--- storage
WARN nvme0n1 has no driver link
OK 2 physical block devices
The check looked for /sys/block/<disk>/device/driver. That is right for
a SCSI or ATA disk, whose parent is the SCSI device carrying the driver
link, but the parent of an NVMe namespace is the controller -- a class
device nothing binds a driver to -- and the driver sits one level
further down on the PCI function:
/sys/block/nvme0n1/device -> ../../nvme0
/sys/block/nvme0n1/device/device/driver -> ../../../../bus/pci/drivers/nvme
So walk up from the disk's parent instead of looking at that one
directory, at most three levels so that a genuinely unbound device
still reports nothing rather than its host controller's driver.
On sigusr2 running 6.18.51.mx64.498 the check now says
--- storage
nvme0n1 driven by nvme
sda driven by sd
OK 2 physical block devices
and sd is still reported as before on a SCSI-only machine.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
donald
reviewed
Sep 16, 2026
| def release_key(release): | ||
| """Sort key for `6.12.100.mx64.493': the mariux revision decides ties.""" | ||
| numbers = [int(part) for part in re.findall(r"\d+", release)] | ||
| return tuple(numbers + [0] * (5 - len(numbers)))[:5] |
Collaborator
There was a problem hiding this comment.
This sorts "6.12.1-mx64.493 < "6.12-mx64.492" because (6, 12, 1, 64, 493) < (6, 12, 64, 492, 0). It also sorts "6.12.2-mx64.493" < "6.12-rc4-mx64.492" because (6, 12, 2, 64, 493)) < (6, 12, 4, 64, 492).
Collaborator
There was a problem hiding this comment.
Maybe not important because releases are looked at below?
Collaborator
|
Nice! |
Sign in
to join this conversation on GitHub.
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Smoke-test the kernel a machine is running.
linux.be0 configures with
make olddefconfig, which answers every new ornewly visible symbol with its default and never asks, so a symbol that
upstream renames, splits or re-parents silently turns off -- and the module
is simply gone from the built kernel. Going from 6.12 to 6.18 that cost us
several drivers, and nothing noticed until the hardware they drive was used.
make oldconfigwould have asked, but it cannot be used unattended.This script therefore looks at the running kernel the way that regression
shows up in practice: hardware that no longer has a driver, filesystems and
network features that no longer work, modules that the reference kernel had
and this one has not. It is meant to be run on a machine that has been
rebooted into the new kernel, before the new kernel is rolled out further.
--vm goes further and boots the kernel package in qemu, on any machine and
without rebooting anything: an initramfs of this machine’s own binaries, and
a guest that loads the modules, filters packets with iptables, makes and
mounts ext4, XFS, squashfs and overlayfs, and puts a veth pair across two
network namespaces. It takes about ten seconds, and --vm-sweep adds every
module in the tree, compared against the kernel this one replaces so that a
module which did not load before the update is not blamed on the update.
Everything works as an ordinary user; the checks that need root say so and
are skipped. Run it as root as well if you can, which adds the checks that
actually load modules and mount filesystems. The VM needs /dev/kvm to be
usable to be quick, and qemu; it needs no privileges.
FAIL is a defect in the kernel package: something the site needs is missing.
WARN depends on the machine rather than on the package (hardware absent, a
device deliberately unbound, a buffer already wrapped) and needs a look.
SKIP is a check this machine cannot answer.