Skip to content

nvidiactl: Manage Vulkan EGL loader JSON #578

Jump to bottom
Merged
merged 7 commits into from
Sep 11, 2026
Merged

nvidiactl: Manage Vulkan EGL loader JSON #578

merged 7 commits into from
Sep 11, 2026

Conversation

@pmenzel
Copy link
Contributor

@pmenzel pmenzel commented Sep 2, 2026 •
edited
Loading

Create the symbolic links below:

/etc/vulkan/icd.d                → /usr/share/mxgfx/vulkan/icd.d
/etc/vulkan/implicit_layer.d     → /usr/share/mxgfx/vulkan/implicit_layer.d
/etc/glvnd/egl_vendor.d          → /usr/share/mxgfx/glvnd/egl_vendor.d
/etc/egl/egl_external_platform.d → /usr/share/mxgfx/egl/egl_external_platform.d

Tested on sigusr2.

Intentionally left commits in, that were reverted in the end.

pmenzel added 7 commits August 28, 2026 07:51
@pmenzel
nvidiactl: Manage Vulkan and GLVND-EGL loader JSON on GPU selection
1b6868e 
Create the NVIDIA Vulkan ICD, implicit layer, and GLVND-EGL vendor JSON
files under /etc only when an NVIDIA GPU is selected, and remove them when
falling back to the generic mxgfx link. This prevents the Vulkan/EGL
loaders on generic or nouveau hosts from trying to dlopen the then
dangling NVIDIA libraries.

Fixes: mariux64/mariux64-issues#161
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
@pmenzel
nvidiactl: Remove elaborate comment
118fdd7
@pmenzel
nvidiactl: Link the selected driver's Vulkan and EGL loader JSON
94a04e1 
Commit 1b6868ec8801 ("nvidiactl: Manage Vulkan and GLVND-EGL loader JSON on
GPU selection") hardcoded the JSON taken from NVIDIA-Linux-x86_64-570.144,
assuming the content is version independent. It is not. Comparing the files
in the beebuilds below /src/mariux/beeroot/build-archives:

    driver        api_version   file_format_version   nvidia_layers.json
    510.60.02     1.3.194       1.0.0                 "layer": { }
    510.108.03    1.3.194       1.0.0                 "layer": { }
    535.104.05    1.3.242       1.0.0                 "layer": { }
    535.230.02    1.3.242       1.0.0                 "layer": { }
    570.144       1.4.303       1.0.1                 "layers": [ { } ]
    580.178.04    1.4.312       1.0.1                 "layers": [ { }, { } ]

Only 10_nvidia.json is identical across all six. 580.178.04 also adds a
second layer, VK_LAYER_NV_present, whose library_path is the fully versioned
libnvidia-present.so.580.178.04 rather than a bare soname, so the hardcoded
copy silently dropped it.

Since /usr/share/nvidia/kernel/<kver>/current selects any of those six
drivers depending on the booted kernel, one hardcoded copy is wrong on most
hosts -- it overclaimed api_version 1.4.303 on 510 and 535 nodes and
underclaimed it on 580 nodes.

Let 'nvidia_current' ship the files verbatim below /usr/share/nvidia/<version>,
which is on no loader search path, and only link the selected version into
/etc here. A file is linked when it exists, so drivers packaged before that
change get nothing rather than a dangling link and this can be deployed
before every driver is rebuilt.

Link: mariux64/bee-files#3429
Resolves: mariux64/mariux64-issues#161
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
@pmenzel
nvidiactl: Also link the EGL external-platform JSON
d13f75b 
Since bee-files PR 3429, 'nvidia_current' ships the Wayland, GBM, Xlib and XCB
external-platform JSON together with their libraries, but nothing puts them on
a loader search path. Link them like the rest, the existing guard skipping the
two that 510.60.02 does not have.

Test on a node with an NVIDIA GPU selected:

    for l in $(grep -Rh library_path /etc/{vulkan,glvnd,egl} | cut -d'"' -f4 | sort -u) ; do echo "$l: $(readlink -e "$(ldconfig -p | awk -v l=$l '$1==l{print $NF;exit}')" || echo NOT FOUND)" ; done

Link: mariux64/bee-files#3429
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
@pmenzel
install.sh: Never let install_symlink descend into its link name
ffa33f6 
'ln -sfv target link' treats 'link' as a directory to create the link *in*
whenever it is a directory, or a symbolic link to one. install_symlink()
therefore silently installs to the wrong place as soon as the link name is
already occupied by a directory:

    $ mkdir -p a/icd.d
    $ ln -sfv /usr/share/mxgfx/vulkan/icd.d a/icd.d
    'a/icd.d/icd.d' -> '/usr/share/mxgfx/vulkan/icd.d'

That is the case for any /etc directory a tool created at runtime before
mxtools started shipping it as a link, and the readlink() guard above does
not catch it -- readlink() of a directory is empty, so it always differs
from the wanted target.

'--no-dereference' is not enough, it only covers a symbolic link to a
directory, not a real one. Use '--no-target-directory', which makes both
cases behave:

    $ ln -sfTv /usr/share/mxgfx/vulkan/icd.d a/icd.d
    ln: a/icd.d: cannot overwrite directory

Fail the install rather than continue, as the file is registered in the bee
CONTENT either way, and a wrongly placed link would be recorded as if it
had been installed.

Assisted-by: Claude Opus 5 <noreply@anthropic.com>
@pmenzel
nvidiactl: Stop managing the loader JSON below /etc
a5f42d7 
Writing the Vulkan, GLVND-EGL and EGL external-platform JSON links into /etc
at runtime does not fit how mariux hosts are maintained:

* pdist deletes them again. With --delete it collects every entry of each
  distributed directory (add_clean_dir()) and removes what the master does
  not have, so the nightly dist takes /etc/vulkan, /etc/glvnd and /etc/egl
  off every NVIDIA node until the next reboot. An EXCEPTS entry would have
  to live in /root/Distfile on the master, invisible to anyone reading
  mxtools.

* The reverse holds too: were the master ever to have an NVIDIA GPU, its
  links would be pushed to every host, and there they dangle as file links
  -- the noisy case, as the loader open()s them and complains.

* The links outlive the GPU. Every no-GPU path does call
  remove_nvidia_loader_json(), so a disk moved into a card-less PC cleans
  itself on the next boot, but not if nvidia.service is masked or dies
  early, and nvidia.service has no ExecStop at all.

Drop it. The following commit puts the very same directories on the loader
search path with constant symbolic links shipped by mxtools, which is
pdist-neutral and leaves no state behind. Both mechanisms must not coexist:
set_generic_mxgfx_link() called remove_nvidia_loader_json() *before*
flipping /node/usr_share_mxgfx, so with the directories linked,

    rm -f /etc/vulkan/icd.d/nvidia_icd.json

would still resolve into /usr/share/nvidia/<version>/ and delete the JSON
out of the driver package. Today only the read-only /usr prevents it.

This reverts the nvidiactl part of commits 1b6868ec8801 ("nvidiactl: Manage
Vulkan and GLVND-EGL loader JSON on GPU selection"), 94a04e1
("nvidiactl: Link the selected driver's Vulkan and EGL loader JSON") and
d13f75b ("nvidiactl: Also link the EGL external-platform JSON"),
leaving the file as it was before the series.

Link: mariux64/mariux64-issues#161
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
@pmenzel
etc: Link the loader JSON directories to the selected graphics driver
b485698 
Commit a5f42d7cbd2b ("nvidiactl: Stop managing the loader JSON below /etc")
took the runtime linking out. Put the same directories on the loader search
path statically instead, pointing at the mxgfx tree that nvidiactl already
switches:

    /etc/vulkan/icd.d                -> /usr/share/mxgfx/vulkan/icd.d
    /etc/vulkan/implicit_layer.d     -> /usr/share/mxgfx/vulkan/implicit_layer.d
    /etc/glvnd/egl_vendor.d          -> /usr/share/mxgfx/glvnd/egl_vendor.d
    /etc/egl/egl_external_platform.d -> /usr/share/mxgfx/egl/egl_external_platform.d

/node/usr_share_mxgfx thus stays the single switch: it points to
/usr/share/nvidia/<version>, where 'nvidia_current' ships the JSON since
bee-files PR 3429, and the loaders see exactly the files of the driver
selected for the booted kernel. On generic and nouveau nodes it points to
mxgfx-generic, which has no such directories, and the links do not resolve.

Being constant and installed by mxtools, they are on the master as well and
pdist distributes them like any other file -- no EXCEPTS entry, no drift, no
state that can outlive the GPU.

The paths are compiled into the loaders and /etc comes first, so nothing
that lives below /usr/share is displaced:

    $ strings /usr/lib/libEGL.so.1 | grep vendor.d
    /etc/glvnd/egl_vendor.d:/usr/share/glvnd/egl_vendor.d
    $ strings /usr/share/nvidia/580.178.04/lib/libEGL_nvidia.so.580.178.04 | grep external_platform.d
    /etc/egl/egl_external_platform.d:/usr/share/egl/egl_external_platform.d

A link that does not resolve is the quiet case: opendir() returns the very
ENOENT that today's missing directory returns, and the remaining directory
is scanned as before. On a generic node, with the link installed to a
DESTDIR:

    $ strace -e trace=openat eglinfo -B 2>&1 | grep -E 'egl_vendor|_mesa.json'
    openat(AT_FDCWD, "/etc/glvnd/egl_vendor.d", ...|O_DIRECTORY) = -1 ENOENT (No such file or directory)
    openat(AT_FDCWD, "/usr/share/glvnd/egl_vendor.d", ...|O_DIRECTORY) = 3
    openat(AT_FDCWD, "/usr/share/glvnd/egl_vendor.d/50_mesa.json", O_RDONLY) = 3

    $ __EGL_VENDOR_LIBRARY_DIRS=$DESTDIR/etc/glvnd/egl_vendor.d:/usr/share/glvnd/egl_vendor.d \
        strace -e trace=openat eglinfo -B 2>&1 | grep -E 'egl_vendor|_mesa.json'
    openat(AT_FDCWD, "…/destdir/etc/glvnd/egl_vendor.d", ...|O_DIRECTORY) = -1 ENOENT (No such file or directory)
    openat(AT_FDCWD, "/usr/share/glvnd/egl_vendor.d", ...|O_DIRECTORY) = 3
    openat(AT_FDCWD, "/usr/share/glvnd/egl_vendor.d/50_mesa.json", O_RDONLY) = 3

The leaf directories are linked rather than /etc/{vulkan,glvnd,egl}, so
/etc/vulkan stays a real directory and a loader settings file below
/etc/vulkan/settings.d remains possible -- /etc is the only loader search
path that is writable where /usr is a read-only shared image.

Nodes on which the previous, runtime variant already ran carry
/etc/vulkan/icd.d and friends as real directories. install_symlink() now
refuses those loudly instead of installing into them, see commit
ffa33f6d76b6 ("install.sh: Never let install_symlink descend into its link
name"); remove them once by hand. pdist heals them on its own, it moves a
directory aside where the master has a symbolic link.

Test results Firefox 155.0.1:

1.  about:support shows:

    > WebGL 1 Driver Renderer NVIDIA Corporation -- NVIDIA GeForce RTX 2080/PCIe/SSE2
    > WebGL 1 Driver Version  3.2.0 NVIDIA 580.178.04

2.  Frames per second for 1024 x 1024:

     5000: 57 to 52
    20000: 48 to 50
    25000: 41
    30000: 35

Link: mariux64/bee-files#3429
Resolves: mariux64/mariux64-issues#161
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
@pmenzel pmenzel force-pushed the nvidiactl-manage-vulkan-egl-loader-json branch from 820e28a to b485698 Compare September 11, 2026 19:56
@pmenzel pmenzel merged commit d7af506 into master Sep 11, 2026
pmenzel added a commit that referenced this pull request Sep 18, 2026
@pmenzel
nvidiactl: Disable GSP firmware via hostconfig tag *nvidia-no-gsp*
0d0945b 
Driver 580.178.04 and earlier enable the GPU System Processor (GSP)
firmware by default on Turing cards.  Since hooking up [Nvidia’s EGL
libraries][1], on *ploetze* (OptiPlex 7071, RTX 2080/TU104) this leads
to a reproducible lockup cascade under Xorg load: Xid 38/8 channel
errors, repeated Xid 109 (CTX SWITCH TIMEOUT), until the GSP firmware
itself panics (Xid 120, RISC-V crash report) and stops answering RPCs
(Xid 119).  The GPU is then unrecoverable until reboot.

    [580377.250202] NVRM: GPU at PCI:0000:01:00: GPU-0cd39e0b-595f-07c2-e5b3-ef9a66092d64
    [580377.257777] NVRM: Xid (PCI:0000:01:00): 62, 00011fab 00012007 00011b38 000159fb 00015e06 00013e17 00000011 00000000
    [580383.269003] NVRM: Xid (PCI:0000:01:00): 119, Timeout after 6s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) (0x20800a56 0x5c).
    [580383.283524] NVRM: GPU0 GSP RPC buffer contains function 76 (GSP_RM_CONTROL) and data 0x0000000020800a56 0x000000000000005c.
    [580383.294732] NVRM: GPU0 RPC history (CPU -> GSP):
    [580383.299440] NVRM:     entry function                   data0              data1              ts_start           ts_end             duration actively_polling
    [580383.313504] NVRM:      0    76   GSP_RM_CONTROL        0x0000000020800a56 0x000000000000005c 0x00065b93db1410f3 0x0000000000000000          y
    [580383.326278] NVRM:     -1    76   GSP_RM_CONTROL        0x00000000c3700104 0x0000000000000014 0x00065b93dad6badd 0x00065b93dad6bced    528us
    [580383.339045] NVRM:     -2    76   GSP_RM_CONTROL        0x00000000c3700104 0x0000000000000014 0x00065b93dad6acb0 0x00065b93dad6ae57    423us
    [580383.351812] NVRM:     -3    76   GSP_RM_CONTROL        0x00000000c3700104 0x0000000000000014 0x00065b93dad69e94 0x00065b93dad6a177    739us
    [580383.364579] NVRM:     -4    76   GSP_RM_CONTROL        0x00000000c3700104 0x0000000000000014 0x00065b93dad69754 0x00065b93dad69cca   1398us
    [580383.377344] NVRM:     -5    76   GSP_RM_CONTROL        0x00000000c3700104 0x0000000000000014 0x00065b93dad68d89 0x00065b93dad68efb    370us
    [580383.390120] NVRM:     -6    76   GSP_RM_CONTROL        0x00000000c3700104 0x0000000000000014 0x00065b93dad67e73 0x00065b93dad683a3   1328us
    [580383.402898] NVRM:     -7    76   GSP_RM_CONTROL        0x00000000c3700104 0x0000000000000014 0x00065b93dad64e4b 0x00065b93dad651fe    947us
    [580383.415673] NVRM: GPU0 RPC event history (CPU <- GSP):
    [580383.420898] NVRM:     entry function                   data0              data1              ts_start           ts_end             duration during_incomplete_rpc
    [580383.435394] NVRM:      0    4130 RECOVERY_ACTION       0x0000000000000000 0x0000000000000000 0x00065b93db1410b0 0x00065b93db1410b1      1us
    [580383.448160] NVRM:     -1    4102 OS_ERROR_LOG          0x0000000000000000 0x0000000000000000 0x00065b93db13c9fc 0x00065b93db1410af  18099us
    [580383.460935] NVRM:     -2    4128 GSP_POST_NOCAT_RECORD 0x0000000000000003 0x0000000000011fab 0x00065b93db13c9fa 0x00065b93db13c9fc      2us
    [580383.473703] NVRM:     -3    4128 GSP_POST_NOCAT_RECORD 0x0000000000000005 0x00000285057f2624 0x00065b937828a266 0x00065b937828a267      1us
    [580383.486467] NVRM:     -4    4128 GSP_POST_NOCAT_RECORD 0x0000000000000002 0x0000000000000025 0x00065b93780dc3f3 0x00065b93780dc3f8      5us
    [580383.499231] NVRM:     -5    4099 POST_EVENT            0x0000000000000001 0x0000000000000000 0x00065b93780dc3db 0x00065b93780dc3f1     22us
    [580383.511995] NVRM:     -6    4128 GSP_POST_NOCAT_RECORD 0x0000000000000005 0x00000285057f2624 0x00065b873abd27cf 0x00065b873abd27d0      1us
    [580383.524761] NVRM:     -7    4128 GSP_POST_NOCAT_RECORD 0x0000000000000005 0x00000285057f2624 0x00065b873abc9137 0x00065b873abc9139      2us
    [580383.537527] CPU: 9 UID: 0 PID: 711 Comm: nv_queue Tainted: P           O       6.12.74.mx64.489 #1
    [580383.537530] Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
    [580383.537530] Hardware name: Dell Inc. OptiPlex 7071/097YXY, BIOS 1.2.1 11/11/2019
    [580383.537531] Call Trace:
    [580383.537533]  <TASK>
    [580383.537534]  dump_stack_lvl+0x53/0x70
    [580383.537539]  _nv013207rm+0x508/0x5b0 [nvidia]

Pass `NVreg_EnableGpuFirmware=0` to insmod on hosts carrying the new
hostconfig tag *nvidia-no-gsp*, so the resource manager runs on the CPU
again, as it did on Turing before the 580 branch.  Off by default:
other GSP-capable desktops (RTX A4000, RTX 3060, RTX 4090) have not
reported crashes, so only affected hosts (*ploetze* now, the identically
equipped pcpool machines when needed) get the tag in `/etc/hostconfig`.

insmod does not consult `/etc/modprobe.d`, so the parameter has to be
given on the command line.  Probe the module with `modinfo` first,
because `insmod` fails hard on unknown parameters and drivers before 510
do not have this one.

[1]: #578

Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Fixes: mariux64/mariux64-issues#229
pmenzel added a commit that referenced this pull request Sep 18, 2026
@pmenzel
nvidiactl: Disable GSP firmware via hostconfig tag *nvidia-no-gsp*
0dd6b95 
Driver 580.178.04 and earlier enable the GPU System Processor (GSP)
firmware by default on Turing cards.  Since hooking up [Nvidia’s EGL
libraries][1], on *ploetze* (OptiPlex 7071, RTX 2080/TU104) this leads
to a reproducible lockup cascade under Xorg load: Xid 38/8 channel
errors, repeated Xid 109 (CTX SWITCH TIMEOUT), until the GSP firmware
itself panics (Xid 120, RISC-V crash report) and stops answering RPCs
(Xid 119).  The GPU is then unrecoverable until reboot.

    [580377.250202] NVRM: GPU at PCI:0000:01:00: GPU-0cd39e0b-595f-07c2-e5b3-ef9a66092d64
    [580377.257777] NVRM: Xid (PCI:0000:01:00): 62, 00011fab 00012007 00011b38 000159fb 00015e06 00013e17 00000011 00000000
    [580383.269003] NVRM: Xid (PCI:0000:01:00): 119, Timeout after 6s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) (0x20800a56 0x5c).
    [580383.283524] NVRM: GPU0 GSP RPC buffer contains function 76 (GSP_RM_CONTROL) and data 0x0000000020800a56 0x000000000000005c.
    [580383.294732] NVRM: GPU0 RPC history (CPU -> GSP):
    [580383.299440] NVRM:     entry function                   data0              data1              ts_start           ts_end             duration actively_polling
    [580383.313504] NVRM:      0    76   GSP_RM_CONTROL        0x0000000020800a56 0x000000000000005c 0x00065b93db1410f3 0x0000000000000000          y
    [580383.326278] NVRM:     -1    76   GSP_RM_CONTROL        0x00000000c3700104 0x0000000000000014 0x00065b93dad6badd 0x00065b93dad6bced    528us
    [580383.339045] NVRM:     -2    76   GSP_RM_CONTROL        0x00000000c3700104 0x0000000000000014 0x00065b93dad6acb0 0x00065b93dad6ae57    423us
    [580383.351812] NVRM:     -3    76   GSP_RM_CONTROL        0x00000000c3700104 0x0000000000000014 0x00065b93dad69e94 0x00065b93dad6a177    739us
    [580383.364579] NVRM:     -4    76   GSP_RM_CONTROL        0x00000000c3700104 0x0000000000000014 0x00065b93dad69754 0x00065b93dad69cca   1398us
    [580383.377344] NVRM:     -5    76   GSP_RM_CONTROL        0x00000000c3700104 0x0000000000000014 0x00065b93dad68d89 0x00065b93dad68efb    370us
    [580383.390120] NVRM:     -6    76   GSP_RM_CONTROL        0x00000000c3700104 0x0000000000000014 0x00065b93dad67e73 0x00065b93dad683a3   1328us
    [580383.402898] NVRM:     -7    76   GSP_RM_CONTROL        0x00000000c3700104 0x0000000000000014 0x00065b93dad64e4b 0x00065b93dad651fe    947us
    [580383.415673] NVRM: GPU0 RPC event history (CPU <- GSP):
    [580383.420898] NVRM:     entry function                   data0              data1              ts_start           ts_end             duration during_incomplete_rpc
    [580383.435394] NVRM:      0    4130 RECOVERY_ACTION       0x0000000000000000 0x0000000000000000 0x00065b93db1410b0 0x00065b93db1410b1      1us
    [580383.448160] NVRM:     -1    4102 OS_ERROR_LOG          0x0000000000000000 0x0000000000000000 0x00065b93db13c9fc 0x00065b93db1410af  18099us
    [580383.460935] NVRM:     -2    4128 GSP_POST_NOCAT_RECORD 0x0000000000000003 0x0000000000011fab 0x00065b93db13c9fa 0x00065b93db13c9fc      2us
    [580383.473703] NVRM:     -3    4128 GSP_POST_NOCAT_RECORD 0x0000000000000005 0x00000285057f2624 0x00065b937828a266 0x00065b937828a267      1us
    [580383.486467] NVRM:     -4    4128 GSP_POST_NOCAT_RECORD 0x0000000000000002 0x0000000000000025 0x00065b93780dc3f3 0x00065b93780dc3f8      5us
    [580383.499231] NVRM:     -5    4099 POST_EVENT            0x0000000000000001 0x0000000000000000 0x00065b93780dc3db 0x00065b93780dc3f1     22us
    [580383.511995] NVRM:     -6    4128 GSP_POST_NOCAT_RECORD 0x0000000000000005 0x00000285057f2624 0x00065b873abd27cf 0x00065b873abd27d0      1us
    [580383.524761] NVRM:     -7    4128 GSP_POST_NOCAT_RECORD 0x0000000000000005 0x00000285057f2624 0x00065b873abc9137 0x00065b873abc9139      2us
    [580383.537527] CPU: 9 UID: 0 PID: 711 Comm: nv_queue Tainted: P           O       6.12.74.mx64.489 #1
    [580383.537530] Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
    [580383.537530] Hardware name: Dell Inc. OptiPlex 7071/097YXY, BIOS 1.2.1 11/11/2019
    [580383.537531] Call Trace:
    [580383.537533]  <TASK>
    [580383.537534]  dump_stack_lvl+0x53/0x70
    [580383.537539]  _nv013207rm+0x508/0x5b0 [nvidia]

Pass `NVreg_EnableGpuFirmware=0` to insmod on hosts carrying the new
hostconfig tag *nvidia-no-gsp*, so the resource manager runs on the CPU
again, as it did on Turing before the 580 branch.  Off by default:
other GSP-capable desktops (RTX A4000, RTX 3060, RTX 4090) have not
reported crashes, so only affected hosts (*ploetze* now, the identically
equipped pcpool machines when needed) get the tag in `/etc/hostconfig`.

insmod does not consult `/etc/modprobe.d`, so the parameter has to be
given on the command line.  Probe the module with `modinfo` first,
because `insmod` fails hard on unknown parameters and drivers before 510
do not have this one.

[1]: #578

Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Resolves: mariux64/mariux64-issues#229
Sign in to join this conversation on GitHub.

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@pmenzel