How to limit the number of open files for nfsd with systemd

Question

It's fairly straightforward to control open file limits for many services with systemd: You create a file at /etc/systemd/system/$service_name.service.d/limits.conf, add a [Service] section with LimitNOFILE=12345, run systemctl daemon-reload, restart your service and double-check /proc/$pid/limits to verify.

However, I'm unable to get open file limits to go beyond the default 1024/4096 for any nfsd process. I have NFS configured for 512 threads and therefore I have 512 nfsd processes running under kthreadd (PID 2).

Here's output of systemctl cat nfs:

# /usr/lib/systemd/system/nfs-server.service
[Unit]
Description=NFS server and services
DefaultDependencies=no
Requires= network.target proc-fs-nfsd.mount
Requires= nfs-mountd.service
Wants=rpcbind.socket network-online.target
Wants=rpc-statd.service nfs-idmapd.service
Wants=rpc-statd-notify.service

After= network-online.target local-fs.target
After= proc-fs-nfsd.mount rpcbind.socket nfs-mountd.service
After= nfs-idmapd.service rpc-statd.service
Before= rpc-statd-notify.service

# GSS services dependencies and ordering
Wants=auth-rpcgss-module.service
After=rpc-gssd.service gssproxy.service

Wants=nfs-config.service
After=nfs-config.service

[Service]
EnvironmentFile=-/run/sysconfig/nfs-utils

Type=oneshot
RemainAfterExit=yes
ExecStartPre=-/usr/sbin/exportfs -r
ExecStartPre=-/bin/sh -c '/bin/kill -HUP `cat /run/gssproxy.pid`'
ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS
ExecStop=/usr/sbin/rpc.nfsd 0
ExecStopPost=/usr/sbin/exportfs -au
ExecStopPost=/usr/sbin/exportfs -f

ExecReload=-/usr/sbin/exportfs -r

[Install]
WantedBy=multi-user.target

# /run/systemd/generator/nfs-server.service.d/order-with-mounts.conf
# Automatically generated by nfs-server-generator

[Unit]
RequiresMountsFor=/mnt/files
# /etc/systemd/system/nfs-server.service.d/override.conf
[Service]
LimitNOFILE=infinity

As you can see, I'm trying to set LimitNOFILE to infinity (cat /proc/sys/fs/file-max produces 7171636).

However:

# cat /proc/$(pidof -s nfsd)/limits | grep files
Max open files            1024                 4096                 files

Also, SELinux is disabled.

This is Amazon Linux 2 LTS Candidate:

# cat /proc/version
Linux version 4.9.62-10.57.amzn2.x86_64 (mockbuild@ip-10-0-1-120) (gcc version 7.2.1 20170915 (Red Hat 7.2.1-2) (GCC) ) #1 SMP Wed Dec 6 00:07:49 UTC 2017

Does anyone know where this per-process open file limit is coming from? And how do I change it?

Answer 1

From man rpc.nfsd:

The rpc.nfsd program implements the user level part of the NFS service. The main functionality is handled by the nfsd kernel module. The user space program merely specifies what sort of sockets the kernel service should listen on, what NFS versions it should support, and how many kernel threads it should use.

Since the actual nfsd processes are started by the kernel (via kthreadd as you've noted), they are not (even nth order grand-)children of systemd and so systemd has no way of passing increased ulimit values to them.

Answer 2

I may have found something that could be at least a useful hint, though it isn't really a complete answer. If you want to change the limit on the number of files that can be opened for the NFS process, you can run this:

echo -n "Max open files=32768:65535" > /proc/<<THE NFS PID>>/limits

This will change the limit for the running process, but this may not actually be what you want. I'm having trouble with "Too many open files" errors on NFS, and the above command doesn't get rid of those errors. I presume that the above command only increases the number of files that the NFS daemon itself can open, rather than making it possible for other processes that are using NFS to open more files. Anyway, hope this ends up being useful to someone at some point in the future.