Unable to connect to certain IP address, but able to connect to other IP in same subnet

Question

I'm running a 14.04.2 LTS ubuntu box in a DMZ with IP of 10.10.30.35 and I have the following network mystery. I'm trying to connect to another unix box outside the DMZ but in our internal network at 10.2.0.200 (via ssh or https) and failing with errors like No route to host or ping brings Destination Host Unreachable. However, I can connect (ping, ssh, & https) to a similar unix box at 10.2.0.170 that is also outside the DMZ but inside our network. I have nothing in my iptables:

> iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

The gateway looks to be set up correctly:

> route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.10.30.25     0.0.0.0         UG    0      0        0 eth0
10.10.30.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0

When I do a tcpdump from the machine in the DMZ and try to ping the 10.2.0.200 box, all I ever see is this:

 ARP, Request who-has 10.2.0.200 tell myserver.mydomain.com, length 28

However, I never see any response or any other traffic to 10.2.0.200. So it looks like the arp protocol is not working? Checking arp, I get this:

> arp -na
? (outsideipaddress) at <incomplete> on eth0
? (10.10.30.25) at 00:1a:8c:f0:50:82 [ether] on eth0 <-- gateway
? (10.10.30.80) at 00:50:56:a7:06:89 [ether] on eth0
? (outsideipaddress) at <incomplete> on eth0
? (10.2.0.200) at <incomplete> on eth0
? (outsideipaddress) at <incomplete> on eth0

So, that 10.2.0.200 entry is odd, I try to clear it out and get this:

> arp -d 10.2.0.200
SIOCDARP(dontpub): Network is unreachable

Attempts to use ip -s -s neigh flush all do not remove the entry either.

So, I am at my wits end here. Is it the arp entry that is preventing my connection to 10.2.0.200? Or something else I am missing altogether? Can I just manually edit the arp table somehow and add the MAC address?

Thanks in advance for your help.

Answer 1

I did a

ip route flush cache

and I was able to ping and SSH. Still can't https, but I believe that is the network firewall. Thank you all for the help.

Answer 2

The ? (10.2.0.200) at <incomplete> on eth0 entry is a normal behavior when the machine cannot be reached (is the machine powered on?)

If the destination machine has something in the IPTables, for example, it's blocking the ICMPv4 request, you will see a complete entry in the arp table because the traffic was discarted but it has reached to the host.

So, just guessing. Do you have a firewall doing the DMZ? Just for be sure... try with a trace route.