Skip to content

Add linux 5.15 #2748

Merged
merged 10 commits into from
Oct 7, 2022
Merged

Add linux 5.15 #2748

merged 10 commits into from
Oct 7, 2022

Conversation

donald
Copy link
Collaborator

@donald donald commented Sep 20, 2022

Latest kernel of latest LTR

  • Includes several security fixes ( All from my dvteam mail 23.8 )
  • NR_CPUS = 512
  • Includes overlayfs (for e.g. rootless docker)
  • Includes fixes to shut up ext4 and loop device messages ( so that mxqd tmpdir cleanup can be made faster via device mapper and ext4 shutdown ioctl)
  • Includes a long list of new, interesting, undiscovered bugs.

Tested on theinternet (with nvidia)

Create bee file with

    cp linux-5.10.113-434.bee linux-5.15.69-435.bee

and update SRCURL.
Create bee file with

  cp nvidia_linux-5.10.113-434-510.60.02-0.bee nvidia_linux-5.15.69-435-510.60.02-0.bee
@donald
Copy link
Collaborator Author

donald commented Sep 20, 2022

[  168.689025] general protection fault, probably for non-canonical address 0x2571ffeffbf38: 0000 [#1] SMP PTI
[  168.698803] CPU: 8 PID: 1061 Comm: xfdesktop Tainted: P           O      5.15.69.mx64.435 #1
[  168.707269] Hardware name: Dell Inc. Precision Tower 5810/0K240Y, BIOS A08 07/31/2015
[  168.715121] RIP: 0010:queued_spin_lock_slowpath+0x167/0x1a0
[  168.720715] Code: f3 90 48 8b 32 48 85 f6 74 f6 eb d9 c1 ee 12 83 e0 03 83 ee 01 48 c1 e0 04 48 63 f6 48 05 40 b8 02 00 48 03 04 f5 60 68 2e 82 <48> 89 10 8b 42 08 85 c0 75 09 f3 90 8b 42 08 85 c0 74 f7 48 8b 32
[  168.739540] RSP: 0018:ffffc9000170bcb0 EFLAGS: 00010202
[  168.744783] RAX: 0002571ffeffbf38 RBX: ffff88811a190080 RCX: 0000000000240000
[  168.751938] RDX: ffff88883fc2b840 RSI: 0000000000003bff RDI: ffff88812548e878
[  168.759093] RBP: ffff88812548e87c R08: 0000000000240000 R09: 0000000000000000
[  168.766248] R10: 0000000000000000 R11: 0000000000000008 R12: 0000000000000000
[  168.773401] R13: 0000000000000002 R14: ffff88812548e878 R15: ffff88812548e870
[  168.780558] FS:  00007f0e7a115980(0000) GS:ffff88883fc00000(0000) knlGS:0000000000000000
[  168.788674] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  168.794435] CR2: 00007f3a128f9443 CR3: 0000000115408002 CR4: 00000000001706e0
[  168.801590] Call Trace:
[  168.804044]  <TASK>
[  168.806148]  __mutex_lock.constprop.0+0x163/0x490
[  168.810875]  ? nfs_inode_attach_open_context+0x80/0x80 [nfs]
[  168.816568]  ? obj_cgroup_charge_pages+0x92/0x1a0
[  168.821294]  nfs_scan_commit+0x3a/0xb0 [nfs]
[  168.825596]  __nfs_commit_inode+0x108/0x180 [nfs]
[  168.830331]  ? nfs_inode_attach_open_context+0x80/0x80 [nfs]
[  168.836018]  nfs_wb_all+0x59/0x110 [nfs]
[  168.839969]  nfs4_inode_return_delegation+0x58/0x90 [nfsv4]
[  168.845585]  nfs4_proc_remove+0x101/0x110 [nfsv4]
[  168.850320]  nfs_unlink+0xf8/0x2d0 [nfs]
[  168.854265]  vfs_unlink+0x10e/0x280
[  168.857768]  do_unlinkat+0x19e/0x2c0
[  168.861355]  __x64_sys_unlink+0x3e/0x60
[  168.865203]  ? __ia32_sys_unlink+0x60/0x60
[  168.869310]  do_syscall_64+0x43/0x90
[  168.872902]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[  168.877973] RIP: 0033:0x7f0e7cceeaa7
[  168.881559] Code: f0 ff ff 73 01 c3 48 8b 0d be 03 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 91 03 0d 00 f7 d8 64 89 01 48
[  168.900384] RSP: 002b:00007ffcb4db6d98 EFLAGS: 00000202 ORIG_RAX: 0000000000000057
[  168.907975] RAX: ffffffffffffffda RBX: 0000000000827b00 RCX: 00007f0e7cceeaa7
[  168.915129] RDX: 000000000046d018 RSI: 0000000000000007 RDI: 00000000008b1970
[  168.922282] RBP: 00000000008b1970 R08: 00000000008ab130 R09: 000000000046d012
[  168.929436] R10: 00007f0e7cc070a8 R11: 0000000000000202 R12: 00000000004e1300
[  168.936589] R13: 00000000008a7190 R14: 000000000080ac30 R15: 00007f0e7cf1b960
[  168.943743]  </TASK>
[  168.945932] Modules linked in: rpcsec_gss_krb5 nfsv4 nfs nvidia_drm(PO) nvidia_modeset(PO) nvidia_uvm(PO) 8021q garp stp mrp llc nvidia(PO) drm_kms_helper fb_sys_fops syscopyarea sysfillrect sysimgblt drm x86_pkg_temp_thermal snd_usb_audio uvcvideo videobuf2_vmalloc videobuf2_memops snd_usbmidi_lib videobuf2_v4l2 snd_hwdep kvm_intel videobuf2_common wmi_bmof snd_rawmidi snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi snd_hda_intel snd_intel_dspcfg kvm snd_hda_codec input_leds snd_hda_core led_class snd_pcm snd_timer irqbypass e1000e snd crc32c_intel iTCO_wdt iTCO_vendor_support soundcore wmi nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc ip_tables x_tables unix ipv6 autofs4
[  169.009275] ---[ end trace a0f8c7ec2dc318ec ]---
[  169.013909] RIP: 0010:queued_spin_lock_slowpath+0x167/0x1a0
[  169.019497] Code: f3 90 48 8b 32 48 85 f6 74 f6 eb d9 c1 ee 12 83 e0 03 83 ee 01 48 c1 e0 04 48 63 f6 48 05 40 b8 02 00 48 03 04 f5 60 68 2e 82 <48> 89 10 8b 42 08 85 c0 75 09 f3 90 8b 42 08 85 c0 74 f7 48 8b 32
[  169.038326] RSP: 0018:ffffc9000170bcb0 EFLAGS: 00010202
[  169.043564] RAX: 0002571ffeffbf38 RBX: ffff88811a190080 RCX: 0000000000240000
[  169.050716] RDX: ffff88883fc2b840 RSI: 0000000000003bff RDI: ffff88812548e878
[  169.057869] RBP: ffff88812548e87c R08: 0000000000240000 R09: 0000000000000000
[  169.065021] R10: 0000000000000000 R11: 0000000000000008 R12: 0000000000000000
[  169.072175] R13: 0000000000000002 R14: ffff88812548e878 R15: ffff88812548e870
[  169.079328] FS:  00007f0e7a115980(0000) GS:ffff88883fc00000(0000) knlGS:0000000000000000
[  169.087439] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  169.093200] CR2: 00007f3a128f9443 CR3: 0000000115408002 CR4: 00000000001706e0

later a process was hanging with

buczek@theinternet:~$ sudo cat /proc/5692/stack 
[<0>] wait_on_commit+0x71/0xb0 [nfs]
[<0>] __nfs_commit_inode+0x131/0x180 [nfs]
[<0>] nfs_wb_all+0x59/0x110 [nfs]
[<0>] nfs4_inode_return_delegation+0x58/0x90 [nfsv4]
[<0>] nfs4_proc_remove+0x101/0x110 [nfsv4]
[<0>] nfs_unlink+0xf8/0x2d0 [nfs]
[<0>] vfs_unlink+0x10e/0x280
[<0>] do_unlinkat+0x19e/0x2c0
[<0>] __x64_sys_unlink+0x3e/0x60
[<0>] do_syscall_64+0x43/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x61/0xcb

@donald
Copy link
Collaborator Author

donald commented Sep 21, 2022

And again (from a new boot)

[52132.511523] general protection fault, probably for non-canonical address 0x2120782520682869: 0000 [#1] SMP PTI
[52132.521522] CPU: 0 PID: 367 Comm: thunderbird Tainted: P           O      5.15.69.mx64.435 #1
[52132.530041] Hardware name: Dell Inc. Precision Tower 5810/0K240Y, BIOS A08 07/31/2015
[52132.537863] RIP: 0010:osq_lock+0x4d/0x100
[52132.541868] Code: 10 00 00 00 00 48 c7 02 00 00 00 00 89 42 14 87 07 85 c0 0f 84 a2 00 00 00 83 e8 01 48 98 48 03 0c c5 60 68 2e 82 48 89 4a 08 <48> 89 11 44 8b 42 10 45 85 c0 0f 85 81 00 00 00 65 48 8b 34 25 00
[52132.560631] RSP: 0018:ffffc900017b3cb0 EFLAGS: 00010206
[52132.565852] RAX: ffffffffffff8881 RBX: ffffc900017b3d60 RCX: 2120782520682869
[52132.572983] RDX: ffff88883fa2b800 RSI: ffff8882614bfbc0 RDI: ffff8882614bfbd4
[52132.580113] RBP: ffff8882614bfbd4 R08: 0000000000000001 R09: 0000000000000000
[52132.587242] R10: 0000000000000000 R11: 61c8864680b583eb R12: 0000000000000000
[52132.594373] R13: 0000000000000002 R14: 0000000000000001 R15: ffff8882614bfbc8
[52132.601502] FS:  00007f15063c1740(0000) GS:ffff88883fa00000(0000) knlGS:0000000000000000
[52132.609588] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[52132.615329] CR2: 00007f14e0c1e000 CR3: 0000000260770006 CR4: 00000000001706f0
[52132.622461] Call Trace:
[52132.624905]  <TASK>
[52132.627000]  __mutex_lock.constprop.0+0xd6/0x490
[52132.631617]  ? inode_permission.part.0+0x146/0x190
[52132.636406]  nfs_scan_commit+0x3a/0xb0 [nfs]
[52132.640682]  __nfs_commit_inode+0x108/0x180 [nfs]
[52132.645387]  nfs_wb_all+0x59/0x110 [nfs]
[52132.649313]  nfs4_inode_return_delegation+0x58/0x90 [nfsv4]
[52132.654893]  nfs4_proc_remove+0x101/0x110 [nfsv4]
[52132.659597]  nfs_unlink+0xf8/0x2d0 [nfs]
[52132.663521]  vfs_unlink+0x10e/0x280
[52132.667008]  do_unlinkat+0x19e/0x2c0
[52132.670581]  __x64_sys_unlink+0x3e/0x60
[52132.674412]  ? __ia32_sys_unlink+0x60/0x60
[52132.678503]  do_syscall_64+0x43/0x90
[52132.682076]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[52132.687127] RIP: 0033:0x7f15064b1aa7
[52132.690698] Code: f0 ff ff 73 01 c3 48 8b 0d be 03 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 91 03 0d 00 f7 d8 64 89 01 48
[52132.709463] RSP: 002b:00007ffceb17bcb8 EFLAGS: 00000202 ORIG_RAX: 0000000000000057
[52132.717028] RAX: ffffffffffffffda RBX: 00007f15061f59a0 RCX: 00007f15064b1aa7
[52132.724159] RDX: 0000000000000003 RSI: 0000000000000003 RDI: 00007f15061faee0
[52132.731288] RBP: 0000000000000000 R08: 0000000000000000 R09: 00007f14cd1008d8
[52132.738417] R10: 0000000000000001 R11: 0000000000000202 R12: 00007ffceb17bd80
[52132.745546] R13: 00007ffceb17d010 R14: 00007f1506190848 R15: 0000000000000000
[52132.752677]  </TASK>
[52132.754858] Modules linked in: rpcsec_gss_krb5 nfsv4 nfs nvidia_drm(PO) nvidia_modeset(PO) nvidia_uvm(PO) 8021q garp stp mrp llc nvidia(PO) drm_kms_helper fb_sys_fops syscopyarea sysfillrect sysimgblt drm snd_usb_audio uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_usbmidi_lib snd_hwdep videobuf2_common snd_rawmidi wmi_bmof x86_pkg_temp_thermal snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi kvm_intel snd_hda_intel snd_intel_dspcfg snd_hda_codec kvm snd_hda_core snd_pcm input_leds irqbypass led_class e1000e crc32c_intel iTCO_wdt snd_timer iTCO_vendor_support snd soundcore wmi nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc ip_tables x_tables unix ipv6 autofs4
[52132.817942] ---[ end trace 93cb7f590ebf1f4b ]---
[52132.822557] RIP: 0010:osq_lock+0x4d/0x100
[52132.826566] Code: 10 00 00 00 00 48 c7 02 00 00 00 00 89 42 14 87 07 85 c0 0f 84 a2 00 00 00 83 e8 01 48 98 48 03 0c c5 60 68 2e 82 48 89 4a 08 <48> 89 11 44 8b 42 10 45 85 c0 0f 85 81 00 00 00 65 48 8b 34 25 00
[52132.845350] RSP: 0018:ffffc900017b3cb0 EFLAGS: 00010206
[52132.850573] RAX: ffffffffffff8881 RBX: ffffc900017b3d60 RCX: 2120782520682869
[52132.857705] RDX: ffff88883fa2b800 RSI: ffff8882614bfbc0 RDI: ffff8882614bfbd4
[52132.864848] RBP: ffff8882614bfbd4 R08: 0000000000000001 R09: 0000000000000000
[52132.871980] R10: 0000000000000000 R11: 61c8864680b583eb R12: 0000000000000000
[52132.879122] R13: 0000000000000002 R14: 0000000000000001 R15: ffff8882614bfbc8
[52132.886255] FS:  00007f15063c1740(0000) GS:ffff88883fa00000(0000) knlGS:0000000000000000
[52132.894342] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[52132.900096] CR2: 00007f14e0c1e000 CR3: 0000000260770006 CR4: 00000000001706f0

and yet again

[  265.043237] BUG: unable to handle page fault for address: 0000000000050e5c
[  265.050119] #PF: supervisor read access in kernel mode
[  265.055254] #PF: error_code(0x0000) - not-present page
[  265.060391] PGD 8000000107d73067 P4D 8000000107d73067 PUD 122f32067 PMD 0 
[  265.067264] Oops: 0000 [#1] SMP PTI
[  265.070753] CPU: 2 PID: 1843 Comm: geany Tainted: P           O      5.15.69.mx64.435 #1
[  265.078843] Hardware name: Dell Inc. Precision Tower 5810/0K240Y, BIOS A08 07/31/2015
[  265.086671] RIP: 0010:__mutex_lock.constprop.0+0xbb/0x490
[  265.092071] Code: 41 5e 41 5f c3 cc cc cc cc 65 48 8b 04 25 00 ad 01 00 48 8b 00 a8 08 0f 85 94 00 00 00 49 8b 07 48 83 e0 f8 0f 84 4d 03 00 00 <8b> 50 34 85 d2 74 7b 8b 40 48 e8 26 a5 62 ff 49 8d 6f 0c 48 89 ef
[  265.110843] RSP: 0018:ffffc90002153cb8 EFLAGS: 00010206
[  265.116065] RAX: 0000000000050e28 RBX: ffffc90002153d60 RCX: ffff88811767e040
[  265.123196] RDX: ffff88811767e040 RSI: 0000000000050e28 RDI: ffff888123e38730
[  265.130329] RBP: ffffc90002153d50 R08: 0000000000000001 R09: 0000000000000000
[  265.137460] R10: 0000000000000000 R11: 61c8864680b583eb R12: 0000000000000000
[  265.144592] R13: 0000000000000002 R14: 0000000000000001 R15: ffff888123e38730
[  265.151722] FS:  00007fb5c24cd980(0000) GS:ffff88885f480000(0000) knlGS:0000000000000000
[  265.159809] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  265.165551] CR2: 0000000000050e5c CR3: 000000011ed18006 CR4: 00000000001706e0
[  265.172683] Call Trace:
[  265.175127]  <TASK>
[  265.177224]  ? inode_permission.part.0+0x146/0x190
[  265.182019]  nfs_scan_commit+0x3a/0xb0 [nfs]
[  265.186299]  __nfs_commit_inode+0x108/0x180 [nfs]
[  265.191009]  nfs_wb_all+0x59/0x110 [nfs]
[  265.191019]  nfs4_inode_return_delegation+0x58/0x90 [nfsv4]
[  265.191036]  nfs4_proc_remove+0x101/0x110 [nfsv4]
[  265.191045]  nfs_unlink+0xf8/0x2d0 [nfs]
[  265.209154]  vfs_unlink+0x10e/0x280
[  265.212642]  do_unlinkat+0x19e/0x2c0
[  265.216217]  __x64_sys_unlink+0x3e/0x60
[  265.220049]  ? __ia32_sys_unlink+0x60/0x60
[  265.224142]  do_syscall_64+0x43/0x90
[  265.227719]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[  265.232770] RIP: 0033:0x7fb5c519baa7
[  265.236344] Code: f0 ff ff 73 01 c3 48 8b 0d be 03 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 91 03 0d 00 f7 d8 64 89 01 48
[  265.255115] RSP: 002b:00007ffd1bf4bda8 EFLAGS: 00000286 ORIG_RAX: 0000000000000057
[  265.262684] RAX: ffffffffffffffda RBX: 00007ffd1bf4bdb0 RCX: 00007fb5c519baa7
[  265.269817] RDX: 00000000000001ff RSI: 00007ffd1bf4bdb0 RDI: 00000000004d4680
[  265.276948] RBP: 0000000000000002 R08: 00007fb5c24cdc88 R09: 0000000000405010
[  265.284082] R10: 00007fb5c50b40a8 R11: 0000000000000286 R12: 00007ffd1bf4c208
[  265.291214] R13: 0000000000000008 R14: 00000000004d4680 R15: 00000000004dcb30
[  265.298347]  </TASK>
[  265.300528] Modules linked in: rpcsec_gss_krb5 nfsv4 nfs nvidia_drm(PO) nvidia_modeset(PO) nvidia_uvm(PO) 8021q garp stp mrp llc nvidia(PO) drm_kms_helper fb_sys_fops syscopyarea sysfillrect sysimgblt drm x86_pkg_temp_thermal uvcvideo snd_usb_audio kvm_intel videobuf2_vmalloc videobuf2_memops snd_usbmidi_lib videobuf2_v4l2 snd_hwdep kvm videobuf2_common snd_rawmidi wmi_bmof snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi snd_hda_intel input_leds led_class snd_intel_dspcfg snd_hda_codec snd_hda_core snd_pcm irqbypass iTCO_wdt e1000e crc32c_intel snd_timer iTCO_vendor_support snd soundcore wmi nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc ip_tables x_tables unix ipv6 autofs4
[  265.363641] CR2: 0000000000050e5c
[  265.366980] ---[ end trace 2b717acaff9b35cc ]---
[  265.371601] RIP: 0010:__mutex_lock.constprop.0+0xbb/0x490
[  265.377003] Code: 41 5e 41 5f c3 cc cc cc cc 65 48 8b 04 25 00 ad 01 00 48 8b 00 a8 08 0f 85 94 00 00 00 49 8b 07 48 83 e0 f8 0f 84 4d 03 00 00 <8b> 50 34 85 d2 74 7b 8b 40 48 e8 26 a5 62 ff 49 8d 6f 0c 48 89 ef
[  265.395791] RSP: 0018:ffffc90002153cb8 EFLAGS: 00010206
[  265.401018] RAX: 0000000000050e28 RBX: ffffc90002153d60 RCX: ffff88811767e040
[  265.408159] RDX: ffff88811767e040 RSI: 0000000000050e28 RDI: ffff888123e38730
[  265.415300] RBP: ffffc90002153d50 R08: 0000000000000001 R09: 0000000000000000
[  265.422438] R10: 0000000000000000 R11: 61c8864680b583eb R12: 0000000000000000
[  265.429575] R13: 0000000000000002 R14: 0000000000000001 R15: ffff888123e38730
[  265.436712] FS:  00007fb5c24cd980(0000) GS:ffff88885f480000(0000) knlGS:0000000000000000
[  265.444803] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  265.450551] CR2: 0000000000050e5c CR3: 000000011ed18006 CR4: 00000000001706e0

@thomas
Copy link
Collaborator

thomas commented Sep 21, 2022

I can not tell how helpfull it is for the current case. But I had no NFS issues whilst trying an unpatched (nfstop and friends) 5.15.56 kernel on otherland (see messages.2022-08 ibd.).

Use mariux-5.15.69-436 which added upstream commit 6e176d47160c ("NFSv4:
Fixes for nfs4_inode_return_delegation()"):

Author: Trond Myklebust <trond.myklebust@hammerspace.com>
Date:   Sun Oct 10 10:58:12 2021 +0200

    NFSv4: Fixes for nfs4_inode_return_delegation()

    We mustn't call nfs_wb_all() on anything other than a regular file.
    Furthermore, we can exit early when we don't hold a delegation.

    Reported-by: David Wysochanski <dwysocha@redhat.com>
    Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>

diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
index 11118398f495..7c9eb679dbdb 100644
--- a/fs/nfs/delegation.c
+++ b/fs/nfs/delegation.c
@@ -755,11 +755,13 @@ int nfs4_inode_return_delegation(struct inode *inode)
        struct nfs_delegation *delegation;

        delegation = nfs_start_delegation_return(nfsi);
-       /* Synchronous recall of any application leases */
-       break_lease(inode, O_WRONLY | O_RDWR);
-       nfs_wb_all(inode);
-       if (delegation != NULL)
+       if (delegation != NULL) {
+               /* Synchronous recall of any application leases */
+               break_lease(inode, O_WRONLY | O_RDWR);
+               if (S_ISREG(inode->i_mode))
+                       nfs_wb_all(inode);
                return nfs_end_delegation_return(inode, delegation, 1);
+       }
        return 0;
 }
@donald
Copy link
Collaborator Author

donald commented Sep 21, 2022

@pmenzel got a patch from Trond. First test looked good.

I've added that patch to https://github.molgen.mpg.de/mariux64/linux/commits/mariux-5.15.69-436 and will make a new bee package.

From:    Trond Myklebust [<trondmy@hammerspace.com>](mailto:trondmy@hammerspace.com)
Subject: Re: nfs_scan_commit: BUG: unable to handle page fault for address:
 000000001d473c07
To:      ["anna@kernel.org"](mailto:anna@kernel.org) [<anna@kernel.org>](mailto:anna@kernel.org), ["pmenzel@molgen.mpg.de"](mailto:pmenzel@molgen.mpg.de)
	[<pmenzel@molgen.mpg.de>](mailto:pmenzel@molgen.mpg.de)


Hi Paul,

On Wed, 2022-09-21 at 13:42 +0200, Paul Menzel wrote:
> Dear Linux folks,
>
>
> Moving from Linux 5.10.113 to 5.15.69, starting Mozilla Thunderbird
> or 
> Mozilla Firefox with the home on NFS, both programs get killed, and 
> Linux 5.15.69 logs:
>
> ```
> [ 3827.604396] BUG: unable to handle page fault for address: 
> 000000001d473c07
> [ 3827.611297] #PF: supervisor read access in kernel mode
> [ 3827.616452] #PF: error_code(0x0000) - not-present page
> [ 3827.621604] PGD 0 P4D 0
> [ 3827.624152] Oops: 0000 [#1] SMP PTI
> [ 3827.627657] CPU: 0 PID: 2378 Comm: firefox Not tainted 
> 5.15.69.mx64.435 #1
> [ 3827.634551] Hardware name: Dell Inc. Precision Tower 3620/0MWYPT, 
> BIOS 2.20.0 12/09/2021
> [ 3827.642659] RIP: 0010:nfs_scan_commit_list+0x1e/0x100 [nfs]
> [ 3827.648256] Code: 66 66 2e 0f 1f 84 00 00 00 00 00 90 0f 1f 44 00
> 00 
> 41 57 41 56 41 55 41 54 55 53 48 83 ec 10 4c 8b 2f 48 89 3c 24 89 4c
> 24 
> 0c <49> 8b 5d 00 4c 39 ef 0f 84 c3 00 00 00 48 89 f5 49 89 d6 4d 89
> ef
> [ 3827.667057] RSP: 0018:ffffc90002097ce0 EFLAGS: 00010282
> [ 3827.672294] RAX: 000000006329dcd6 RBX: ffffc90002097d60 RCX: 
> 000000007fffffff
> [ 3827.679440] RDX: ffffc90002097d60 RSI: ffffc90002097d50 RDI: 
> ffff8881d7618b38
> [ 3827.686587] RBP: ffffc90002097d50 R08: 0000000000000001 R09: 
> 0000000000000000
> [ 3827.693734] R10: 0000000000000000 R11: 61c8864680b583eb R12: 
> 0000000000000000
> [ 3827.700880] R13: 000000001d473c07 R14: 0000000000000001 R15: 
> 0000000000000000
> [ 3827.708027] FS:  00007fa6141f2780(0000) GS:ffff88881dc00000(0000) 
> knlGS:0000000000000000
> [ 3827.716131] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 3827.721886] CR2: 000000001d473c07 CR3: 000000012dae0006 CR4: 
> 00000000003706f0
> [ 3827.729034] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
> 0000000000000000
> [ 3827.736180] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 
> 0000000000000400
> [ 3827.743328] Call Trace:
> [ 3827.745779]  <TASK>
> [ 3827.747883]  nfs_scan_commit+0x76/0xb0 [nfs]
> [ 3827.752167]  __nfs_commit_inode+0x108/0x180 [nfs]
> [ 3827.756886]  nfs_wb_all+0x59/0x110 [nfs]
> [ 3827.760822]  nfs4_inode_return_delegation+0x58/0x90 [nfsv4]
> [ 3827.766413]  nfs4_proc_remove+0x101/0x110 [nfsv4]
> [ 3827.771130]  nfs_unlink+0xf5/0x2d0 [nfs]
> [ 3827.775065]  vfs_unlink+0x10b/0x280
> [ 3827.778563]  do_unlinkat+0x19e/0x2c0
> [ 3827.782158]  __x64_sys_unlink+0x3e/0x60
> [ 3827.786002]  ? __x64_sys_readlink+0x1b/0x30
> [ 3827.790192]  do_syscall_64+0x40/0x90
> [ 3827.793779]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
> [ 3827.798847] RIP: 0033:0x7fa6142e2aa7
> [ 3827.802435] Code: f0 ff ff 73 01 c3 48 8b 0d be 03 0d 00 f7 d8 64
> 89 
> 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 b8 57 00 00 00
> 0f 
> 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 91 03 0d 00 f7 d8 64 89 01
> 48
> [ 3827.821264] RSP: 002b:00007fff37879a08 EFLAGS: 00000202 ORIG_RAX: 
> 0000000000000057
> [ 3827.828848] RAX: ffffffffffffffda RBX: 0000000080004005 RCX: 
> 00007fa6142e2aa7
> [ 3827.835997] RDX: 0000000077120e8d RSI: 00007fa614383520 RDI: 
> 00007fa605425b88
> [ 3827.843145] RBP: 00007fa605425b88 R08: 00007fff37879add R09: 
> 0000000000000000
> [ 3827.850291] R10: 00007fa614362ae0 R11: 0000000000000202 R12: 
> 0000000077120e8d
> [ 3827.857439] R13: 00007fff37879add R14: 00007fa6141f26c8 R15: 
> 0000000000000065
> [ 3827.864586]  </TASK>
> [ 3827.866776] Modules linked in: rpcsec_gss_krb5 nfsv4 nfs 8021q
> garp 
> stp mrp llc amdgpu snd_hda_codec_realtek snd_hda_codec_generic 
> ledtrig_audio i915 iommu_v2 gpu_sched drm_ttm_helper iosf_mbi ttm 
> drm_kms_helper x86_pkg_temp_thermal kvm_intel drm kvm
> snd_hda_codec_hdmi 
> intel_gtt i2c_algo_bit fb_sys_fops syscopyarea sysfillrect
> snd_hda_intel 
> input_leds led_class snd_intel_dspcfg sysimgblt e1000e snd_hda_codec 
> hid_logitech_hidpp snd_hda_core hid_logitech_dj snd_usb_audio 
> snd_usbmidi_lib snd_hwdep snd_rawmidi snd_pcm snd_timer uvcvideo 
> videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common
> snd 
> wmi_bmof soundcore wmi iTCO_wdt video irqbypass crc32c_intel 
> iTCO_vendor_support nfsd auth_rpcgss oid_registry nfs_acl lockd grace
> sunrpc ip_tables x_tables unix ipv6 autofs4
> [ 3827.935422] CR2: 000000001d473c07
> [ 3827.938745] ---[ end trace d7dc2bc122fe8836 ]---
> ```
>

Does cherry-picking commit 6e176d47160c ("NFSv4: Fixes for
nfs4_inode_return_delegation()") into 5.15.69 from the upstream kernel
tree fix the problem?

8<---------------------------------------------------
From 6e176d47160cec8bcaa28d9aa06926d72d54237c Mon Sep 17 00:00:00 2001
From: Trond Myklebust [<trond.myklebust@hammerspace.com>](mailto:trond.myklebust@hammerspace.com)
Date: Sun, 10 Oct 2021 10:58:12 +0200
Subject: [PATCH] NFSv4: Fixes for nfs4_inode_return_delegation()

We mustn't call nfs_wb_all() on anything other than a regular file.
Furthermore, we can exit early when we don't hold a delegation.

Reported-by: David Wysochanski [<dwysocha@redhat.com>](mailto:dwysocha@redhat.com)
Signed-off-by: Trond Myklebust [<trond.myklebust@hammerspace.com>](mailto:trond.myklebust@hammerspace.com)
---
 fs/nfs/delegation.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
index 11118398f495..7c9eb679dbdb 100644
--- a/fs/nfs/delegation.c
+++ b/fs/nfs/delegation.c
@@ -755,11 +755,13 @@ int nfs4_inode_return_delegation(struct inode *inode)
 	struct nfs_delegation *delegation;
 
 	delegation = nfs_start_delegation_return(nfsi);
-	/* Synchronous recall of any application leases */
-	break_lease(inode, O_WRONLY | O_RDWR);
-	nfs_wb_all(inode);
-	if (delegation != NULL)
+	if (delegation != NULL) {
+		/* Synchronous recall of any application leases */
+		break_lease(inode, O_WRONLY | O_RDWR);
+		if (S_ISREG(inode->i_mode))
+			nfs_wb_all(inode);
 		return nfs_end_delegation_return(inode, delegation, 1);
+	}
 	return 0;
 }

@donald
Copy link
Collaborator Author

donald commented Sep 22, 2022

Installation instruction:

sudo bee install -f linux-5.15.69-436.x86_64 nvidia_linux-5.15.69-436-510.60.02-0.x86_64
sudo mxgrub mariux-5.15.69-436
sudo mxgrub --reboot

Pro-Tip: make sure, you are on the system you think you are. :-)

@thomas
Copy link
Collaborator

thomas commented Sep 22, 2022

There is no more /proc/[PID]/io file in the procfs :(

Most probably caused by 'CONFIG_TASKSTATS is not set', what in turn disables 'TASK_IO_ACCOUNTING'. Is there a replacement available?

@thomas
Copy link
Collaborator

thomas commented Sep 22, 2022

Well, sorry for bothering. I still don't get it. In the 5.10 kernel the overlay-fs is available, what exactly is the new requirement?

@donald
Copy link
Collaborator Author

donald commented Sep 22, 2022

Well, sorry for bothering. I still don't get it. In the 5.10 kernel the overlay-fs is available, what exactly is the new requirement?

Indeed. Don't know why docker didn't enable its overlay2 driver on dose with the 5.10.113 kernel.

@donald
Copy link
Collaborator Author

donald commented Sep 22, 2022

There is no more /proc/[PID]/io file in the procfs :(

Most probably caused by 'CONFIG_TASKSTATS is not set', what in turn disables 'TASK_IO_ACCOUNTING'. Is there a replacement available?

Hmmm. Bug in build system? I don't get it yet.

buczek@dose:/scratch/local/linux (linux-5.15.69-mpi)$ git restore -s linux-5.10.113-mpi config-mpi 
buczek@dose:/scratch/local/linux (linux-5.15.69-mpi)$ egrep 'TASKSTATS|TASK_IO_ACCOUNTING' config-mpi 
CONFIG_TASK_IO_ACCOUNTING=y
buczek@dose:/scratch/local/linux (linux-5.15.69-mpi)$ cp config-mpi .config
buczek@dose:/scratch/local/linux (linux-5.15.69-mpi)$ make olddefconfig
#
# configuration written to .config
#
buczek@dose:/scratch/local/linux (linux-5.15.69-mpi)$ egrep 'TASKSTATS|TASK_IO_ACCOUNTING' .config
# CONFIG_TASKSTATS is not set
buczek@dose:/scratch/local/linux (linux-5.15.69-mpi)$

@donald
Copy link
Collaborator Author

donald commented Sep 23, 2022

TASKSTATS was auto.selected by KVM until mariux64/linux@63b3f96 which landed in v5.14. So we had TASKSTATS although its default is "N".

Rebuild with a new patch to supress sporadic ext4 warnings when mxqd
uses fcntl EXT4_IOC_SHUTDOWN on its job tmpdirs.

Update minor Linux version while we are at it.
@donald donald changed the title Add linux 5.15.69 Add linux 5.15 Sep 28, 2022
@pmenzel
Copy link
Collaborator

pmenzel commented Oct 5, 2022

Looks good to me.

Includes overlayfs (for e.g. rootless docker)

You striked this through. Not needed in the end?

@donald
Copy link
Collaborator Author

donald commented Oct 6, 2022

You striked this through. Not needed in the end?

Its not a 5.15 feature, its available in 5.10 as well.

@donald donald merged commit 35fede5 into master Oct 7, 2022
Sign in to join this conversation on GitHub.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants