linux-4.14.55: Allow to trace nfsd v4 operations

[donald]

From time to time our nfs file servers are overloaded and it is quite
challenging for the admins to track down the source of the problem.

Add a patch to the kernels nfsd module which enables it to log all nfs4
operations via the kernel event trace utility.

This will give us the oportunity to write simple users-space tools
to track down nfs load.

Also rebuild nvidia_linux-4.14.55 for the new kernel.

Patch Description:

Add trace events to all v4 operations. Record client ip for ipv4
clients. Record uid from request credentials.

For READ and WRITE ops, additionally record offset and length request
arguments and device, inode and filename of the affected file.

Prove of concept:

buczek@claptrap:~/nfs-test$ sudo  ./nfstop.pl 
recording nfs trace for 10 seconds
50 events enabled
Top 5 operation count:
          7232 operations by buczek@theinternet.molgen.mpg.de
           280 operations by buczek@sigchld.molgen.mpg.de
            16 operations by root@afk.molgen.mpg.de
            13 operations by root@sigchld.molgen.mpg.de
             1 operations by root@theinternet.molgen.mpg.de

Top 5 users by read :
         33871 bytes read by buczek@sigchld.molgen.mpg.de

Top 5 users by write :
         33894 bytes write by buczek@sigchld.molgen.mpg.de

Top 5 files read:
         25648 bytes read maj(8),min(32):inode(2148619877) : .Xauthority
              ->      25648 bytes by buczek@sigchld.molgen.mpg.de
          8223 bytes read maj(8),min(32):inode(2148747556) : .bash_history
              ->       8223 bytes by buczek@sigchld.molgen.mpg.de

Top 5 files write:
         25648 bytes write maj(8),min(32):inode(2148619875) : .Xauthority-n
              ->      25648 bytes by buczek@sigchld.molgen.mpg.de
          8246 bytes write maj(8),min(32):inode(2148747556) : .bash_history
              ->       8246 bytes by buczek@sigchld.molgen.mpg.de

tested on dose, sigchld, claptrap

[pmenzel]

Very nice.

But, please remember to set the correct permissions for the patch file, and to use a better commit message summary than linux: Rebuild version 4.14.55. Maybe:

linux-4.14.55: Allow to trace NFSv4 operations

Also, I’ll only accept the merge/pull request, if you give a lightning talk about it at the DV-Treffen. ;-)

[thomas]

Like to have a tutorial :)

BTW: Is it ok to put /home/buczek/nfs-test in the PATH

[wwwutz]

I'd suggest to add the PATH in /etc/profile so all users are able to use it

[donald]

I'll use the tool to monitor access to my home an kill everyone who does this. :-)

Userspace-tools are still work in progress but will eventually go to mxtools. Wouldn't make much sense before some fileservers have booted the new kernel.