Number of open sockets in /proc/pid/fd does not agree with netstat

Question

I'm running into a weird socket issue when running my own crawler. It opens and cloees a lot of TCP sockets rapidly due to the protocol design. It's something I have to live with. I'm very sure in code I have closed the socket correctly (verified via strace and debug prints) . Yet somehow I still hit the open socket limit o my system. Tools like Netdat also shows elevated numbers of open sockets. Upon further inspection. I find there's high numbers of socket fds inside /proc/<pid>/fd/. Here's a sample result I run

All commands are executed as root

# ls /proc/248298/fd/ -l | grep socket | wc -l
522

But when I run netstat to figure out which remote the sockets are connected to, while taking in account of system wide TIME_WAIT and CLOSE_WAIT sockets (since netstat won't associate them with my process anymore). The number is much lower.

# netstat -tulnap | egrep '(TIME_WAIT|CLOSE_WAIT|248298)' | wc -l
109

I've tried setting net.ipv4.tcp_tw_reuse to 1 as mitigation with not success.

What's the cause of this? And further more, why my closed sockets are still counted alive? Or is there ways to work around this?

OS: Linux
Distro: Ubuntu 22.04
Kernel: 5.15
CPU: x64

Answer 1

This is known as ephemeral port pressure and can affect high-traffic systems that make lots of connections to various other services, among other, less legitimate network traffic. What ports an operating system reserves for this varies, as does the RFC advice as to what the range should be (compare RFC 6056, RFC 6335).

The easiest knob for this on Linux is net.ipv4.ip_local_port_range which for systems that make lots of connections probably should be made as large as possible:

sysctl -w net.ipv4.ip_local_port_range=1024\ 65535

This may cause problems with other networked services that make use of ports in the 1024 to 65535 range (RPC for NFS, possibly?). Or, to test the effectiveness of other approaches one can make this value deliberately small to make the problematic state easier to reproduce:

sysctl -w net.ipv4.ip_local_port_range=30000\ 30100

A range this low may of course break random services. Using a test virt or providing for console access to the test system may be advisable.

Otherwise there are various knobs to reduce the length of time a connection spends in the various states (you do not list any of the FIN_WAIT* states in your question which could cause your count to be off), though if set too low these may increase the risk various problems such as when a delayed or duplicated packet shows up. This may be unlikely if the remote systems are dropping the packets due to rate limit throttling.