nft port forwarding not working on router

Question

I have a machine that serves both as a router and a server. I have several lxc containers on this machine, and want to expose them to both the LAN and WAN. Following https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/sec-configuring_port_forwarding_using_nftables I was able to successfully access the servers from both WAN and LAN machines, but not localhost/the router-server itself!

Here is the configuration that partially works:

# Created from lxc-net in debian
table inet lxc {
        chain input {
                type filter hook input priority filter; policy accept;
                iifname "lxcbr0" udp dport { 53, 67 } accept
                iifname "lxcbr0" tcp dport { 53, 67 } accept
        }

        chain forward {
                type filter hook forward priority filter; policy accept;
                iifname "lxcbr0" accept
                oifname "lxcbr0" accept
        }
}
# Created from lxc-net in debian
table ip lxc {
        chain postrouting {
                type nat hook postrouting priority srcnat; policy accept;
                ip saddr 10.0.3.0/24 ip daddr != 10.0.3.0/24 counter packets 51 bytes 3745 masquerade
        }
}

# This is what I added
table ip myportforwarding {
        chain prerouting {
                type nat hook prerouting priority dstnat; policy accept;
                tcp dport 8088 dnat to 10.0.3.230
        }

        chain postrouting {
                type nat hook postrouting priority srcnat; policy accept;
                ip daddr 10.0.3.230 masquerade
        }
}

I tried several options from this answer: How to configure port forwarding with nftables for a Minecraft server on Raspberry Pi?

Nothing seemed to work to enable local access to the services on 8088.

Looking at wireshark, access from LAN looks like:

192.168.1.105 -> 192.168.1.1 SYN
10.0.3.1 -> 10.0.3.230 SYN
...

Access from the same machine:

192.168.1.1 -> 192.168.1.1 SYN
192.168.1.1 <- 192.168.1.1 FIN!

I'm not too familar with nft or iptables, so I'm sure there is something I'm missing

Answer 1

Let's look at a part of the Packet flow in Netfilter and General Networking schematic. It was made for iptables but most of it applies for nftables:

It's documented that the nat table is consulted only for packets in conntrack state NEW: packets starting a new flow.

Routed/forwarded traffic arrives from the nat/prerouting hook: that's where new flows will have a chance to be NAT-ed. OP handled this case.

Locally initiated packets (created at the local process bubble in the center) first traverse the nat/output hook, then their answer will come back as usual through the nat/prerouting hook. Leaving aside the fact that the destination is already not changed for the query, as the answer matches the flow that was created before, it's not a packet in NEW state anymore: the nat/prerouting hook will never be consulted for such traffic because it's too late: the only place to do NAT was nat/output.

So for this case where both routed and locally initiated packets should receive the same alteration, rules in nat/prerouting have to be duplicated in nat/output and usually slightly adapted to match the different case.

The adaptation here is about the host reaching itself, so for the routing case where the interface is the loopback (lo) interface, thus adding oif lo to it. Without this filter, any query from the host to anywhere using port 8088 would be redirected to the container, while only the case for the host to itself is intended.

Adding this chain in the already existing ip myportforwarding table will handle it:

        chain output {
                type nat hook output priority dstnat; policy accept;
                tcp dport 8088 oif "lo" dnat to 10.0.3.230
        }

For the little details: a change from nat/output triggers the reroute check part, where the routing stack is told to reconsider the previous routing decision (output interface lo). After reroute check the output interface becomes lxcbr0.