How does one determine the process causing an ARP request?

Question

I have a machine that runs a network intensive application that spawns many processes. I noticed recently that the machine is producing ARP requests looking for an IP address that does not exist. I would like to trace down which process on the box is causing the ARP requests to be generated for the sake of troubleshooting (so I can know which part of the application is looking for this non-existant IP).

IP's have been changed but they aren't important anyhow.

I discovered these ARP requests by running tcpdump on another machine on the same network:

# tcpdump -i eth0 arp -t -n
ARP, Request who-has 1.1.1.100 tell 1.1.1.1, length 46

There isn't meant to be a device with the address 1.1.1.100, so I want to find what process on 1.1.1.1 is looking for it.

I tried using ss -np | grep 1.1.1.100 as well as netstat -np | grep 1.1.1.100 (netstat is considered deprecated in place of ss for those curious, ss has most of the same options and is meant to perform the same functions). Neither of these return any results, likely because ss and netstat list open sockets, and the ARP request would pre-date a socket being created.

So how can I discern what process causes an ARP request?

Answer 1

ss does show you connections that have not yet been resolved by arp. They are in state SYN-SENT. The problem is that such a state is only held for a few seconds then the connection fails, so you may not see it. You could try rapid polling for it with

while ! ss -p state syn-sent | grep 1.1.1.100; do sleep .1; done

One way to extend the time in this state is to set an arbitrary hard-wired mac address for the IP address in your arp table. Then a connect will take over 30 seconds to timeout, and will be easier to see with ss.

For example, with my eth0 at 192.168.1.1

$ socat tcp:192.168.1.100:80 - 
$ arp -i eth0 -n | grep 192.168.1.100
192.168.1.100 (incomplete) eth0

setting the mac address makes the socat easily visible

$ sudo arp -i eth0 -s 192.168.1.100 80:ef:00:ff:ff:ff
$ socat tcp:192.168.1.100:80 - &
$ ss -p state syn-sent
Netid  Recv-Q Send-Q Local Address:Port Peer Address:Port                
tcp    0      1      192.168.1.1:46608 192.168.1.100:http users:(("socat",pid=20230,fd=3))

Answer 2

You can use nethogs program that displays per-process network traffic. But arp won't be generated directly by running process it would be generated by OS. Some program may want to communicate with 1.1.1.100 and this IP doesn't exists in ARP table so ARP packet is being sent by OS to populate mac address table.

Is DHCP server running on 1.1.1.1? I would say that DHCP server probes addresses in the lease range to see which of them are free.

Answer 3

I had this same problem just now. Having tried a few things, I came back to sysdig. This worked nicely for me:

sysdig fd.rip=1.1.1.100

In my case, the IP in question was actually 172.28.210.22, and the output was:

# sysdig fd.rip=172.28.210.22
5987580 15:42:55.952661802 7 dhclient (1232) < sendto res=300 data=...............*............RT..................................................
6318682 15:43:01.237021372 7 dhclient (1232) > sendto fd=6(<4u>172.28.210.22:67->172.28.208.42:68) size=300 tuple=0.0.0.0:68->172.28.210.22:67
6318683 15:43:01.237080305 7 dhclient (1232) < sendto res=300 data=...............*............RT..................................................
6926596 15:43:10.092470330 7 dhclient (1232) > sendto fd=6(<4u>172.28.210.22:67->172.28.208.42:68) size=300 tuple=0.0.0.0:68->172.28.210.22:67
6926608 15:43:10.092541255 7 dhclient (1232) < sendto res=300 data=...............*............RT..................................................
8882391 15:43:31.595024934 7 dhclient (1232) > sendto fd=6(<4u>172.28.210.22:67->172.28.208.42:68) size=300 tuple=0.0.0.0:68->172.28.210.22:67

which clearly showed that this was coming from dhclient.