Redirecting loopback traffic through a router

Question

For some tests purposes, I am running multiple socket-based applications on one machine and want to simulate "real" network conditions between them. I thought the simplest would be to redirect all traffic between the applications through the router my computer is connected to. Intuitively this should perform more or less like multiple computers connected to the same network, and the network itself should be limited by the capabilities of the router (100Mbps) rather than the local inter-machine socket connections.

My machine (192.168.1.101) is connected directly to a router (192.168.1.2) via the interface em1. I tried to do it by adding an ip route. The result of my ip route command is:

default via 192.168.1.2 dev em1  proto static  metric 1024 
192.168.1.0/24 dev em1  proto kernel  scope link  src 192.168.1.101 
192.168.1.101 via 192.168.1.2 dev em1 

This however doesn't seem to have any effect, since pinging localhost returns more or less the same results as pinging my machine indirectly (192.168.1.101), that is around 0.040 ms. At the same time, pinging a different machine in my network results in pings around 0.3-0.5ms.

I've tried traceroute, here's what I get:

$traceroute 192.168.1.200                    # my other computer
traceroute to 192.168.1.200 (192.168.1.200), 30 hops max, 60 byte packets
 1  192.168.1.200 (192.168.1.200)  1.005 ms  0.972 ms  0.954 ms

$traceroute 192.168.1.101
traceroute to 192.168.1.101 (192.168.1.101), 30 hops max, 60 byte packets
 1  localhost.localdomain (192.168.1.101)  0.051 ms  0.014 ms  0.013 ms

$traceroute 127.0.0.1
traceroute to 127.0.0.1 (127.0.0.1), 30 hops max, 60 byte packets
 1  localhost.localdomain (127.0.0.1)  0.053 ms  0.015 ms  0.013 ms

Am I missing something, or perhaps I'm doing it in a completely wrong way?

Answer 1

The problem is that you have a route in your local table that says:

$ ip route show table local
[...]
local 192.168.1.101 dev eth0 scope host
[...]

When sending a packet with [src=192.168.1.101 dst=192.168.1.101], and expecting the router to send that packet back reflected (some will refuse to this kind of thing), you want the outgoing packet to skip that route, but not the packet coming back.

For that you can change the ip rules:

Remove the catch-all rule for the local table.

# ip rule del from all table local

And replace it by one that doesn't do that for the 192.168.1.101->192.168.1.101 packets:

# ip rule add not from 192.168.1.101 to 192.168.1.101 table local pref 0

Then mark the incoming packets with netfilter:

# iptables -t mangle -I PREROUTING -s 192.168.1.101 -d 192.168.1.101 -j MARK --set-mark 1

And tell ip rule to use the local table for those only:

# ip rule add fwmark 1 table local pref 1

(of course, you also need your ip route add to 192.168.1.101 via 192.168.1.2 in your main table)

Answer 2

Here are a couple of solutions that can introduce latency, packet re-ordering and packet drops on *BSD and Linux.

1. Using netfilter to simulate Packet loss

2. Using tc and netem to introduce delays and loss

3. On FreeBSD, use dummynet. Here's an intro.