nvidiactl: Disable GSP firmware via hostconfig tag *nvidia-no-gsp* #587
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Driver 580.178.04 and earlier enable the GPU System Processor (GSP)
firmware by default on Turing cards. Since hooking up [Nvidia’s EGL
libraries][1], on *ploetze* (OptiPlex 7071, RTX 2080/TU104) this leads
to a reproducible lockup cascade under Xorg load: Xid 38/8 channel
errors, repeated Xid 109 (CTX SWITCH TIMEOUT), until the GSP firmware
itself panics (Xid 120, RISC-V crash report) and stops answering RPCs
(Xid 119). The GPU is then unrecoverable until reboot.
[580377.250202] NVRM: GPU at PCI:0000:01:00: GPU-0cd39e0b-595f-07c2-e5b3-ef9a66092d64
[580377.257777] NVRM: Xid (PCI:0000:01:00): 62, 00011fab 00012007 00011b38 000159fb 00015e06 00013e17 00000011 00000000
[580383.269003] NVRM: Xid (PCI:0000:01:00): 119, Timeout after 6s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) (0x20800a56 0x5c).
[580383.283524] NVRM: GPU0 GSP RPC buffer contains function 76 (GSP_RM_CONTROL) and data 0x0000000020800a56 0x000000000000005c.
[580383.294732] NVRM: GPU0 RPC history (CPU -> GSP):
[580383.299440] NVRM: entry function data0 data1 ts_start ts_end duration actively_polling
[580383.313504] NVRM: 0 76 GSP_RM_CONTROL 0x0000000020800a56 0x000000000000005c 0x00065b93db1410f3 0x0000000000000000 y
[580383.326278] NVRM: -1 76 GSP_RM_CONTROL 0x00000000c3700104 0x0000000000000014 0x00065b93dad6badd 0x00065b93dad6bced 528us
[580383.339045] NVRM: -2 76 GSP_RM_CONTROL 0x00000000c3700104 0x0000000000000014 0x00065b93dad6acb0 0x00065b93dad6ae57 423us
[580383.351812] NVRM: -3 76 GSP_RM_CONTROL 0x00000000c3700104 0x0000000000000014 0x00065b93dad69e94 0x00065b93dad6a177 739us
[580383.364579] NVRM: -4 76 GSP_RM_CONTROL 0x00000000c3700104 0x0000000000000014 0x00065b93dad69754 0x00065b93dad69cca 1398us
[580383.377344] NVRM: -5 76 GSP_RM_CONTROL 0x00000000c3700104 0x0000000000000014 0x00065b93dad68d89 0x00065b93dad68efb 370us
[580383.390120] NVRM: -6 76 GSP_RM_CONTROL 0x00000000c3700104 0x0000000000000014 0x00065b93dad67e73 0x00065b93dad683a3 1328us
[580383.402898] NVRM: -7 76 GSP_RM_CONTROL 0x00000000c3700104 0x0000000000000014 0x00065b93dad64e4b 0x00065b93dad651fe 947us
[580383.415673] NVRM: GPU0 RPC event history (CPU <- GSP):
[580383.420898] NVRM: entry function data0 data1 ts_start ts_end duration during_incomplete_rpc
[580383.435394] NVRM: 0 4130 RECOVERY_ACTION 0x0000000000000000 0x0000000000000000 0x00065b93db1410b0 0x00065b93db1410b1 1us
[580383.448160] NVRM: -1 4102 OS_ERROR_LOG 0x0000000000000000 0x0000000000000000 0x00065b93db13c9fc 0x00065b93db1410af 18099us
[580383.460935] NVRM: -2 4128 GSP_POST_NOCAT_RECORD 0x0000000000000003 0x0000000000011fab 0x00065b93db13c9fa 0x00065b93db13c9fc 2us
[580383.473703] NVRM: -3 4128 GSP_POST_NOCAT_RECORD 0x0000000000000005 0x00000285057f2624 0x00065b937828a266 0x00065b937828a267 1us
[580383.486467] NVRM: -4 4128 GSP_POST_NOCAT_RECORD 0x0000000000000002 0x0000000000000025 0x00065b93780dc3f3 0x00065b93780dc3f8 5us
[580383.499231] NVRM: -5 4099 POST_EVENT 0x0000000000000001 0x0000000000000000 0x00065b93780dc3db 0x00065b93780dc3f1 22us
[580383.511995] NVRM: -6 4128 GSP_POST_NOCAT_RECORD 0x0000000000000005 0x00000285057f2624 0x00065b873abd27cf 0x00065b873abd27d0 1us
[580383.524761] NVRM: -7 4128 GSP_POST_NOCAT_RECORD 0x0000000000000005 0x00000285057f2624 0x00065b873abc9137 0x00065b873abc9139 2us
[580383.537527] CPU: 9 UID: 0 PID: 711 Comm: nv_queue Tainted: P O 6.12.74.mx64.489 #1
[580383.537530] Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
[580383.537530] Hardware name: Dell Inc. OptiPlex 7071/097YXY, BIOS 1.2.1 11/11/2019
[580383.537531] Call Trace:
[580383.537533] <TASK>
[580383.537534] dump_stack_lvl+0x53/0x70
[580383.537539] _nv013207rm+0x508/0x5b0 [nvidia]
Pass `NVreg_EnableGpuFirmware=0` to insmod on hosts carrying the new
hostconfig tag *nvidia-no-gsp*, so the resource manager runs on the CPU
again, as it did on Turing before the 580 branch. Off by default:
other GSP-capable desktops (RTX A4000, RTX 3060, RTX 4090) have not
reported crashes, so only affected hosts (*ploetze* now, the identically
equipped pcpool machines when needed) get the tag in `/etc/hostconfig`.
insmod does not consult `/etc/modprobe.d`, so the parameter has to be
given on the command line. Probe the module with `modinfo` first,
because `insmod` fails hard on unknown parameters and drivers before 510
do not have this one.
[1]: #578
Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Resolves: mariux64/mariux64-issues#229
pmenzel
force-pushed
the
nvidiactl-hook-tag-nvidia-no-gsp-up
branch
from
0d0945b to
0dd6b95
Compare
Sign in
to join this conversation on GitHub.
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves: https://github.molgen.mpg.de/mariux64/mariux64-issues/issues/229