mxgrub: kexec: Missing GPU devices

[pmenzel]

On whiterabbit and jabberwocky:

whiterabbit$ less /var/log/mxqd_log
[…]
No devices were found
2026-07-29 14:49:34 +0200 mxqd[1028]: ERROR: gpu-setup init: Protocol error
No devices were found
2026-07-29 14:51:20 +0200 mxqd[1174]: ERROR: gpu-setup init: Protocol error
No devices were found
2026-07-29 14:52:32 +0200 mxqd[1288]: ERROR: gpu-setup init: Protocol error

$ dmesg
[    0.000000] Linux version 6.12.98.mx64.492 (root@otherland.molgen.mpg.de) (gcc (GCC) 12.5.0, GNU ld (GNU Binutils) 2.41) #1 SMP PREEMPT_DYNAMIC Mon Jul 27 12:51:07 CEST 2026
[    0.000000] Command line: root=LABEL=root ro crashkernel=64G-:256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd audit=0 random.trust_cpu=on systemd.unified_cgroup_hierarchy
[    0.000000] BIOS-provided physical RAM map:
[…]
[   28.303007] Disabling lock debugging due to kernel taint
[   28.308766] nvidia: module license taints kernel.
[   28.968749] nvidia-nvlink: Nvlink Core is being initialized, major device number 238

[   28.995236] nvidia 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0028 address=0xfff01000 flags=0x0000]
[   29.008363] nvidia 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0028 address=0xfa66f000 flags=0x0000]
[   29.019190] nvidia 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0028 address=0xfff01000 flags=0x0000]
[   29.029977] nvidia 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0028 address=0xfa66f000 flags=0x0000]
[   29.040848] nvidia 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0028 address=0xfff01000 flags=0x0000]
[   29.051597] nvidia 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0028 address=0xfa66f000 flags=0x0000]
[   29.997646] nvidia 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0028 address=0xfff01000 flags=0x0000]
[   30.008947] nvidia 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0028 address=0xfa66f000 flags=0x0000]
[   30.019724] nvidia 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0028 address=0xfff01000 flags=0x0000]
[   30.030588] nvidia 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0028 address=0xfa66f000 flags=0x0000]
[   31.738997] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  570.144  Thu Apr 10 20:33:29 UTC 2025
[   32.042270] nvidia_uvm: module uses symbols nvUvmInterfaceDisableAccessCntr from proprietary module nvidia, inheriting taint.
[   32.294134] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  570.144  Thu Apr 10 20:03:03 UTC 2025
[   32.344736] [drm] [nvidia-drm] [GPU ID 0x00002100] Loading driver
[   32.350902] [drm] Initialized nvidia-drm 0.0.0 for 0000:21:00.0 on minor 1
[   33.997725] amd_iommu_report_page_fault: 12 callbacks suppressed
[   33.997732] nvidia 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0028 address=0xfff01000 flags=0x0000]
[   34.014254] nvidia 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0028 address=0xfa66f000 flags=0x0000]
[   34.024719] nvidia 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0028 address=0xfff01000 flags=0x0000]
[   34.035188] nvidia 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0028 address=0xfa66f000 flags=0x0000]
[   34.997587] nvidia 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0028 address=0xfff01000 flags=0x0000]
[   35.008097] nvidia 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0028 address=0xfa66f000 flags=0x0000]
[   35.018563] nvidia 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0028 address=0xfff01000 flags=0x0000]
[   35.029028] nvidia 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0028 address=0xfa66f000 flags=0x0000]
[   35.997604] nvidia 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0028 address=0xfff01000 flags=0x0000]
[   36.008100] nvidia 0000:21:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0028 address=0xfa66f000 flags=0x0000]

[pmenzel]

Same on jabberwocky. Updated the issue description.

[pmenzel]

SBR (secondary bus reset) fixes it:

@jabberwocky:~$ cat /sys/bus/pci/devices/0000:21:00.0/reset_method
flr bus
@jabberwocky:~$ echo bus | sudo tee /sys/bus/pci/devices/0000:21:00.0/reset_method
bus
@jabberwocky:~$ echo 0000:21:00.0 | sudo tee /sys/bus/pci/devices/0000:21:00.0/driver/unbind
0000:21:00.0
@jabberwocky:~$ echo 1 | sudo tee /sys/bus/pci/devices/0000:21:00.0/reset
1
@jabberwocky:~$ echo 0000:21:00.0 | sudo tee /sys/bus/pci/drivers/nvidia/bind
0000:21:00.0
@jabberwocky:~$ cat /sys/bus/pci/devices/0000:21:00.0/reset_method
bus
@jabberwocky:~$ echo default | sudo tee /sys/bus/pci/devices/0000:21:00.0/reset_method
default
@jabberwocky:~$ cat /sys/bus/pci/devices/0000:21:00.0/reset_method
flr bus

(FLR is not enough, and does not bring it back, so just selecting bus would be the right thing.)

Anyway, unloading nvidia modules before doing kexec is required according to Opus 4.8.