nftables, masquerade: packets go through wrong outbounding interface

Question

What I am doing is quite simple, I use port forwarding and enable masquerading on the POSTROUTING chain:

table inet nat {
    chain prerouting {
        type nat hook prerouting priority -100; policy accept;
        ip daddr 198.51.100.105 counter dnat to 10.8.0.105 comment "host.example.com"
    }
    
    chain postrouting {
        type nat hook postrouting priority 100; policy accept;
        counter masquerade
    }
}

I have following interfaces on my machine:

1: lo
2: ens18
3: ens19
4: wg0

The routes are:

default via 203.0.113.1 dev ens18 onlink
10.8.0.0/24 dev wg0 proto kernel scope link src 10.8.0.1
198.51.100.0/24 dev ens19 proto kernel scope link src 198.51.100.3
203.0.113.0/24 dev ens18 proto kernel scope link src 203.0.113.134

And the rules are:

0:      from all lookup local
32764:  from all to 198.51.100.0/24 lookup subnets
32765:  from 198.51.100.0/24 lookup subnets
32766:  from all lookup main
32767:  from all lookup default

The output of ip route get 198.51.100.105 is:

local 198.51.100.105 dev lo table local src 198.51.100.3 uid 0
    cache <local>

The output of ip route show table subnets:

default via 198.51.100.1 dev ens19
198.51.100.0/24 dev ens19 scope link src 198.51.100.3

When I now ping the public address 198.51.100.105 from an external VPS or my DSL at home, I can verify that not only the ICMP packet is received, but also replied:

ICMP Request

root@debian:~# tcpdump -i ens19 icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens19, link-type EN10MB (Ethernet), snapshot length 262144 bytes
21:35:51.686208 IP 192.0.2.2 > 198.51.100.105: ICMP echo request, id 14855, seq 1, length 64

ICMP Reply

root@debian:~# tcpdump -i ens18 icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens18, link-type EN10MB (Ethernet), snapshot length 262144 bytes
21:35:38.797502 IP 198.51.100.105 > 192.0.2.2: ICMP echo reply, id 5107, seq 1, length 64

But as you can see, the response comes from the wrong interface. It should be ens19, but uses ens18 and I don't get why.

1. Why is/are the route/rules not honored? The subnet 198.51.100.0/24 doesn't have anything to do with ens18, it is even a different VLAN.

2. Isn't it possible with nftables to force a different outbounding interface for masqueraded packets?

Thanks a lot for any help.

EDIT: The output of ip -br link; ip -4 -br addr; ip -4 route; ip rule; ip rule show table subnets is:

ip -br link; ip -4 -br addr; ip -4 route; ip rule; ip rule show table subnets
lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
ens18            UP             a6:ea:02:1c:XX:XX <BROADCAST,MULTICAST,UP,LOWER_UP>
ens19            UP             ac:71:fe:18:XX:XX <BROADCAST,MULTICAST,UP,LOWER_UP>
wg0              UNKNOWN        <POINTOPOINT,NOARP,UP,LOWER_UP>
lo               UNKNOWN        127.0.0.1/8
ens18            UP             203.0.113.134/24
ens19            UP             198.51.100.3/24 198.51.100.100/24 198.51.100.101/24 198.51.100.102/24 198.51.100.103/24 198.51.100.104/24 198.51.100.105/24
wg0              UNKNOWN        10.8.0.1/24
default via 203.0.113.1 dev ens18 onlink
10.8.0.0/24 dev wg0 proto kernel scope link src 10.8.0.1
198.51.100.0/24 dev ens19 proto kernel scope link src 198.51.100.3
203.0.113.0/24 dev ens18 proto kernel scope link src 203.0.113.134
0:      from all lookup local
32764:  from all to 198.51.100.0/24 lookup subnets
32765:  from 198.51.100.0/24 lookup subnets
32766:  from all lookup main
32767:  from all lookup default
32764:  from all to 198.51.100.0/24 lookup subnets
32765:  from 198.51.100.0/24 lookup subnets

WireGuard

Output of command wg:

interface: wg0
  public key: XrSd2TftIpiL3zhXXX=
  private key: (hidden)
  listening port: 51820

peer: gZ89rFX6DvBtdeuYXXX=
  endpoint: 233.252.0.0:39126
  allowed ips: 10.8.0.0/24
  latest handshake: 20 seconds ago
  transfer: 3.42 MiB received, 4.08 MiB sent

Output of command systemctl status [email protected]:

● [email protected] - WireGuard via wg-quick(8) for wg0
     Loaded: loaded (/lib/systemd/system/[email protected]; enabled; preset: enabled)
     Active: active (exited) since Sat 2023-08-05 23:42:48 CEST; 14h ago
       Docs: man:wg-quick(8)
             man:wg(8)
             https://www.wireguard.com/
             https://www.wireguard.com/quickstart/
             https://git.zx2c4.com/wireguard-tools/about/src/man/wg-quick.8
             https://git.zx2c4.com/wireguard-tools/about/src/man/wg.8
    Process: 690 ExecStart=/usr/bin/wg-quick up wg0 (code=exited, status=0/SUCCESS)
   Main PID: 690 (code=exited, status=0/SUCCESS)
        CPU: 27ms

Aug 05 23:42:48 debian systemd[1]: Starting [email protected] - WireGuard via wg-quick(8) for wg0...
Aug 05 23:42:48 debian wg-quick[690]: [#] ip link add wg0 type wireguard
Aug 05 23:42:48 debian wg-quick[690]: [#] wg setconf wg0 /dev/fd/63
Aug 05 23:42:48 debian wg-quick[690]: [#] ip -4 address add 10.8.0.1/24 dev wg0
Aug 05 23:42:48 debian wg-quick[690]: [#] ip link set mtu 1420 up dev wg0
Aug 05 23:42:48 debian systemd[1]: Finished [email protected] - WireGuard via wg-quick(8) for wg0.

Server: /etc/wireguard/wg0.conf

[Interface]
Address = 10.8.0.1/24
SaveConfig = true
ListenPort = 51820
PrivateKey = 8BspU4XXX=

[Peer]
PublicKey = gZ89rFX6DvBtdeuYXXX=
AllowedIPs = 10.8.0.0/24
Endpoint = 233.252.0.0:58642

Peer: /etc/wireguard/wg0.conf

[Interface]
# Client Private Key
PrivateKey = iO00+qQDXXX=
Address = 10.8.0.107/24

[Peer]
# Server Public Key
PublicKey = XrSd2TftIpiL3zhXXX=
AllowedIPs = 10.8.0.0/24
PersistentKeepalive = 25
Endpoint = 198.51.100.3:51820

Answer 1

Remarks and adjustments (probably caused by obfuscation):

• I'll assume the WireGuard peer uses 10.8.0.105 instead of 10.8.0.107, to match the nftables ruleset.
• 233.252.0.0 would cause problems in a simulation (especially on peer) because it's a multicast address. I'll use 192.0.2.233 below (with no network relation to 192.0.2.2).

---

This question is to solve a problem about forwarding done with a dnat rule and rather than just local traffic. There's also a hidden problem with the WireGuard tunnel envelope also fixed at the end.

The behavior of the routing stack with the ping test (including the actual dnat that happened in prerouting and the un-masquerade that will happen for the reply) can be summarized with these two commands that query the kernel about what route will be used:

# ip route get from 192.0.2.2 iif ens19 to 10.8.0.105
10.8.0.105 from 192.0.2.2 dev wg0 
    cache iif ens19 
# ip route get from 10.8.0.105 iif wg0 to 192.0.2.2
192.0.2.2 from 10.8.0.105 via 203.0.113.1 dev ens18 
    cache iif wg0 

Here one can see the reply uses the wrong interface. The address will be rewritten by the conntrack entry's content: still 198.51.100.105 even if it doesn't appear above.

This one is caused by a missing rule: anything that comes (back) from wg0 should use the table subnets. Fixed with:

ip rule add iif wg0 lookup subnets

This also fixes the case with rp_filter=1 where the first route test above would just fail with RTNETLINK answers: Invalid cross-device link, even if normally one should add the wg0 route in this table too.

giving now:

# ip route get from 10.8.0.105 iif wg0 to 192.0.2.2
192.0.2.2 from 10.8.0.105 via 198.51.100.1 dev ens19 table subnets 
    cache iif wg0 

The ping test will now work correctly.

---

There's an additional somewhat hidden WireGuard envelope routing problem to.

The combination of:

• not having enabled Strict Reverse Path Forwarding (RFC 3704)
• having the peer contact the server first (see the additional issue at the end)
• having (at least) the kernel implementation figure out it should reply with the same source it was initially contacted to

allows WireGuard to somewhat work, so a ping 10.8.0.1 from peer gets a reply and allows any following WireGuard traffic to continue using the same envelope addresses.

When not stating a source address for a local (non-routed) flow, the routing stack has to figure out which one should be used for the given route. This is especially important for UDP where a socket is often kept unbound (ie: having source 0.0.0.0 aka INADDR_ANY). This is not an issue for a TCP server, as the duplicated established socket created after accept(2) is not bound to 0.0.0.0 anymore but to the correct address: it will then present this address to the routing stack. Here, WireGuard uses UDP with INADDR_ANY. In particular it doesn't bind to 198.51.100.3. That means it presents as source 0.0.0.0 and leaves to the kernel's routing stack the resolution of the outgoing source IP address.

If server's WireGuard had been initiating the very first packet (rather than peer doing it), it would have used 203.0.113.134 instead of 198.51.100.3: the routing stack has no specific ip rule for 0.0.0.0: the ip rule 32765: from 198.51.100.0/24 lookup subnets doesn't match and no special policy routing is applied. In the end, the UDP packet leaves as 203.0.113.134 using ens18.

It appears the kernel implementation at least then continues to use the same address it was queried on. That's not to be relied upon, multi-homing with UDP services requires special support (eg: using IP_PKTINFO) from applications because of this.

Sought outcome for WireGuard:

# ip route get from 198.51.100.105 to 192.0.2.233
192.0.2.233 from 198.51.100.105 via 198.51.100.1 dev ens19 table subnets uid 0 
    cache 

Actual outcome at least if it's the first to initiate traffic:

# ip route get from 0.0.0.0 to 192.0.2.233
192.0.2.233 via 203.0.113.1 dev ens18 src 203.0.113.134 uid 0 
    cache 

To really fix the WireGuard tunnel multi-homed routing itself, one can use a per-L4-protocol routing rule:

ip rule add iif lo ipproto udp sport 51820 lookup subnets

(iif lo is a special syntax to mean locally initiated (non-forwarded) traffic, it's not really about the lo interface).

Giving:

# ip route get from 0.0.0.0 ipproto udp sport 51820 to 192.0.2.233
192.0.2.233 via 198.51.100.1 dev ens19 table subnets src 198.51.100.3 uid 0 
    cache 

Despite presenting INADDR_ANY as source, having the UDP source port 51820 now selects the subnets routing table.