why do I find (under /proc/pid//fd) sockets with broken symlinks

Question

I don't think they're closed, because I would expect that after a close() the fd would disappear, but I see a bunch of lingering fd -> socket[xxxxx] entries with broken symlinks which don't seem to be going away quickly. What causes this state?

Answer 1

The text of the symlink destination does not refer to a file, but to an entry in the /proc/net/tcp table that describes each socket using encoded text fields. For example, on my system at the moment I see:

$ ls -l /proc/24724/fd/7
lrwx------ 1 vagrant vagrant 64 Feb 13 15:08 /proc/24724/fd/7 -> socket:[19164451]

Which corresponds to this line from the tcp table:

$ grep 19164451 /proc/net/tcp
 433: 0100007F:C8AA 0100007F:0C8A 01 00000000:00000000 02:00000286 00000000  1000        0 19164451 2 0000000000000000 20 4 1 10 27

Some quick Google searches should help you find numerous resources for decoding these lines. Two examples:

http://www.onlamp.com/pub/a/linux/2000/11/16/LinuxAdmin.html https://stackoverflow.com/questions/5992211/list-of-possible-internal-socket-statuses-from-proc

If you instead want a tool that processes them for you, netstat will do it if you use the -p option to tell it to read all of your /proc fd links to learn which processes belong to which sockets. Try:

netstat -tuapn

Answer 2

The socket[xxxxx] symlinks are always broken. There is no path you can access in order to open a socket with a given inode number.

I tried it, and it seems you cannot open the file /proc/pidX/fd/Y which refers to a socket. However you can if it refers to a real file, even if the file has been deleted. It is not exactly a symlink. It is magic - i.e. a special case.