Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions mxgrub/mxgrub
Original file line number Diff line number Diff line change
Expand Up @@ -373,10 +373,27 @@ sub cmd_initramfs {
sys('bash','-c','cd /project/admin/initramfs ; find . -name ".git*" -prune -or -print | cpio -H newc -o | gzip > /boot/grub/initramfs.igz');
}

# NVIDIA GSP-based GPUs (e.g. A100) are not reset by kexec: the GSP and its
# WPR2 carveout survive into the next kernel, whose freshly-initialised IOMMU
# then rejects the stale DMA (AMD-Vi IO_PAGE_FAULT), the GSP faults (Xid 120),
# and the driver cannot re-init (RmInitAdapter 0x62 = NV_ERR_RESET_REQUIRED),
# so nvidia-smi reports "No devices were found". Unloading the driver first
# lets it shut the GSP down cleanly so the next kernel enumerates a healthy
# GPU. Best effort: if a module is absent or busy, proceed with the reboot
# anyway (no worse than not unloading at all).
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No idea, if we want long comments, so LLMs better understand the code, or if they should be more terse or even removed.

sub quiesce_nvidia {
return if system('lsmod | grep -q "^nvidia "'); # no NVIDIA driver loaded
warn "mxgrub: unloading NVIDIA driver before kexec ...\n";
for my $mod (qw(nvidia_drm nvidia_modeset nvidia_uvm nvidia)) {
system('rmmod',$mod); # ignore failures (not loaded / in use)
}
}

sub cmd_reboot {
my $chosen=get_chosen();
my $image=label_to_image($chosen);
-e "/boot/$image" or die "/boot/$image: no such file\n";
quiesce_nvidia();
sys('sync');
sys('kexec',"/boot/$image",'--initrd=/boot/grub/initramfs.igz',"--command-line=root=LABEL=root $KERNEL_PARAMETER");
}
Expand Down