While the problem might appear simple, it's nothing but. Having Docker around always imposes several challenges to other parts in the system dealing with networking. Once nftables gets more widely adopted and gets directly used by Docker in the future, and especially once Docker stops using br_netfilter, things might become simpler.
If you think it's still worth using nftables along Docker, I present below a method intended to let Docker handle its part and not requiring having to duplicate Docker settings in other firewall rules whenever changes as simple as starting a new container with a new exposed port, are done.
Problems to be solved
iptables is still needed
Currently (2021) Docker still uses iptables and only iptables (It could also use firewalld but only with firewalld with an iptables backend. I'm not considering this case anyway). There's thus currently no way to have a pure nftables system when using Docker. The fact that iptables can be iptables-legacy or iptables-nft doesn't really matter.
Here are a few relevant excerpts from Docker and iptables that are useful for this case:
Docker installs two custom iptables chains named DOCKER-USER and
DOCKER, and it ensures that incoming packets are always checked by
these two chains first.
All of Docker’s iptables rules are added to the DOCKER chain. Do not
manipulate this chain manually. If you need to add rules which load before
Docker’s rules, add them to the DOCKER-USER chain. These rules are
applied before any rules Docker creates automatically.
Nitpicking: actually Docker does -A DOCKER-USER -j RETURN so rules should be added in it before starting docker, or better: inserted which works in all cases.
Rules added to the FORWARD chain -- either manually, or by another
iptables-based firewall -- are evaluated after these chains.
Docker also sets the policy for the FORWARD chain to DROP. If your
Docker host also acts as a router, this will result in that router not
forwarding any traffic anymore.
Docker enables IP forwarding but firewalls it for other uses than itself by default.
It is possible to set the iptables key to false in the Docker engine’s
configuration file at /etc/docker/daemon.json, but this option is not
appropriate for most users. It is not possible to completely prevent
Docker from creating iptables rules, and creating them after-the-fact
is extremely involved and beyond the scope of these instructions.
Setting iptables to false will more than likely break container
networking for the Docker engine.
Can't avoid having iptables.
Moreover, Docker also loads the kernel module br_netfilter in order to have this property set:
# sysctl net.bridge.bridge-nf-call-iptables
net.bridge.bridge-nf-call-iptables = 1
So bridged frames (with here IPv4 type frames temporarily converted into IPv4 packets) are filtered by iptables and also by nftables (even if this is not clearly documented, nftables just like iptables hooks into Netfilter and Netfilter will call these hooks, be they from iptables or nftables).
This feature is the major point causing problems when interacting with Docker. Without knowledge about it, one would wonder why containers in the same internal bridged LAN can't communicate between themselves anymore, be they handled by Docker or something else (LXC, libvirt/QEMU...) running along Docker.
Here's the Packet flow in Netfilter and General Networking:
A single chain from iptables or nftables in the ip/inet families can thus be traversed in two different ways: from usual routing path (green boxes inside green Network Layer field) but also from bridge path (green boxes inside blue Link Layer field). This documentation also tells:
A bridged packet never enters any network code above layer 1 (Link
Layer). So, a bridged IP packet/frame will never enter the IP code.
So there's a guarantee that a packet won't traverse twice the same chain which is a relief.
Interactions between iptables and nftables
Since the goal is to use nftables one has to know how to use them together.
Here are Q/A having an answer of mine about this:
To summarize:
- iptables and nftables can be used together
- nftables can have its priority adjusted to have a deterministic order of evaluation between iptables and nftables (for this case: nftables after iptables)
- a dropped packet stays definitively dropped whenever/wherever this happens
- an accepted packet (which will be by iptables) continues evaluation in the next chain in the same hook (which will be nftables' chain).
- packet marks can be used to convey messages between iptables and nftables
Method to solve this in a generic way
Dealing with bridge path
nftables rules in the ip/inet families should avoid doing anything in bridge path. Without Docker activating br_netfilter this would never even have to be considered. Detecting being in the bridge path from the ip/inet family should be left to iptables to avoid having nftables to deal with this and stay generic, with Docker installed or not. It's also easier to do this with iptables than with nftables from the ip/inet family because there's the specific iptables -m physdev --physdev-is-bridged test:
[!] --physdev-is-bridged
Matches if the packet is being bridged and therefore is not being routed. This is only useful in the FORWARD and POSTROUTING chains.
Note that this match depends on and loads br_netfilter if this wasn't done by Docker already: to work around issues caused by br_netfilter, br_netfilter is needed!
Using marks to link iptables and nftables
The idea is to use a mark to have messages from iptables passed to nftables, to differentiate cases:
rules evaluation is happening in the bridge path instead of the routing path
Always accept such case.
packet was ACCEPT-ed by Docker
Further restrictions can be added but mostly accept such case.
packet was ignored by Docker
Use normal nftables rules that don't have to consider the presence of Docker.
packet was DROP-ed for any reason in iptables
That's a moot case, nftables won't see this packet and nothing has to or can be done about it.
iptables
If done before Docker is started, create the filter chain DOCKER-USER:
iptables -N DOCKER-USER
If done after, Docker will have created it.
Add a rule to mark packets before Docker evaluation in the DOCKER chain overriden by the bridge path detection case with a different mark (inserting them here as explained before, but numbering them to preserve natural order, which does matter here):
iptables -I DOCKER-USER 1 -j MARK --set-mark 0xd0cca5e
iptables -I DOCKER-USER 2 -m physdev --physdev-is-bridged -j MARK --set-mark 0x10ca1
0x10ca1 and 0xd0cca5e are arbitrarily chosen values.
Append (before or after Docker was run, effect is the same since Docker always inserts its DOCKER chain before) a final rule that resets the packet's mark only if it was the tentative Docker evaluation mark, and add a final ACCEPT rule to override Docker's default DROP policy set on the FORWARD chain: the idea is to defer further evaluation to nftables for packets unrelated to Docker.
iptables -A FORWARD -m mark --mark 0xd0cca5e -j MARK --set-mark 0
iptables -A FORWARD -j ACCEPT
nftables
Change the inet filter forward priority value to a value slightly greater than NF_IP_PRI_FILTER (0), for example 10 to ensure nftables's forward chain happens after iptables filter/FORWARD in order to respect this chronology. The base chain line in OP's ruleset should be changed from:
chain forward {
type filter hook forward priority 0; policy drop;
to:
chain forward {
type filter hook forward priority 10; policy drop;
The 4 previous described cases can be detected in nftables by checking the mark on the packet. Adding counter expressions to help debug.
mark 0x10ca1 : bridge path
Add bridge path pass-through rule:
nft add rule inet filter forward meta mark 0x10ca1 counter accept
mark 0xd0cca5e: Docker case
create a regular/user chain to treat the Docker case and add a rule calling it:
nft add chain inet filter dockercase
nft add rule inet filter forward meta mark 0xd0cca5e counter jump dockercase
add additional restrictions about Docker, but accept by default
For example to restrict incoming packets arriving from the eno2 interface to only be accepted if from private address within 192.168.0.0/16:
nft add rule inet filter dockercase iif eno2 ip saddr != 192.168.0.0/16 counter drop
nft add rule inet filter dockercase counter accept
no mark: general case not related to Docker
Add anything that would be done without having to consider the presence of Docker, including nothing and having a default drop policy, else probably starting with the usual ct state related,established accept
(no packet: dropped in iptables, non-case)
example above becomes:
...
chain forward {
type filter hook forward priority 10; policy drop;
meta mark 0x10ca1 counter accept
meta mark 0xd0cca5e counter jump dockercase
}
chain dockercase {
iif eno2 ip saddr != 192.168.0.0/16 counter drop
counter accept
}
...
Generic handling achieved
The ports 80 and 6200 don't have to appear in the nftables rules anymore. Should a new container that needs to expose new ports be added using Docker commands, nothing at all has to be done in nftables: it's already being taken care of thanks to the marks.
Adding more chains
Still because of br_netfilter's effects, should any other base nftables chain with the property hook forward or hook postrouting contain dropping rules or more usefully, altering rules (nat...) without using tricks described in previous link below figure 7b, then the same kind of arrangement has to be done:
its priority value should be above the iptables' equivalent chain priority
such iptables equivalent chain (except filter/FORWARD where it's already done in DOCKER-USER) should receive:
iptables -t foo -I BAR -m physdev --physdev-is-bridged -j MARK --set-mark 0x10ca1
with foo among raw, mangle, or nat and BAR among PREROUTING or POSTROUTING depending on the case
and the very first rule of the nftables chain should be again:
meta mark 0x10ca1 accept
and if the chain's policy is again drop it should probably again include an user/regular chain jump from a rule using the 0xd0cca5e mark, as previously done.
For hook prerouting, documentation about --physdev-is-bridged tells this might not work in PREROUTING: don't ever use a default drop policy there. Anyway for hook prerouting cases there can't either be any 0xd0cca5e mark inherited from filter/FORWARD yet, but the same would be true using only iptables: PREROUTING can't foresee what happens later.
If you really want to do something at the bridge level, just use nftables in the bridge family, don't rely on this special case of the ip/inet family called from bridge path because of br_netfilter.
Caveat
Now marks are used to handle this, it becomes more difficult to use marks simultaneously for something else, but not impossible with some care. For example by using bitwise operations and masks with these marks. This is available in iptables and nftables. Even ip rule accepts a mask when using a mark as selector.
Important additional required adjustments
Docker adds nat rules to do port forwarding with iptables' DNAT target. In the end all exposed/published ports are routed to the containers instead of being received by the host. That means they will use as seen above iptables's filter/FORWARD chain as well as (with OP's ruleset) nftables inet filter forward chain and won't use INPUT / input.
There are also missing rules preventing correct connectivity for the host.
inet filter input
The input path won't be used at all for Docker's containers, except maybe for the docker-proxy case which usually is for local host's access but that OP already accepts with iif lo accept, so it doesn't have to be further handled in this answer. Anything about Docker shouldn't be present here: references to container's ports 80 and 6200 become useless and should be removed.
Then, unrelated to Docker, the input chain misses a stateful rule. Without it, return traffic from host's output (DNS query replies, ping replies, download for upgrades...) will fail. Use this:
chain input {
type filter hook input priority 0; policy drop;
ct state related,established accept
iif lo accept
iif eno2 icmp type echo-request accept
iif eno2 ip 192.168.0.0/16 tcp dport 22 accept
iif eno2 ip 192.168.0.0/16 tcp dport 443 accept
}
The input path can still require additional rules for Docker itself (rather than its containers): rules might be needed to allow remote access to the Docker API (if security considerations allow it) or various features like VxLAN used by Docker swarm.
inet filter output
Likewise, OP's inet filter output chain's drop policy kills host connectivity (DNS queries, ping requests or downloads can't be initiated etc.). There should either be a policy accept, or the exceptions for needed outgoing traffic from host itself should be added. The chain should include at least something like this:
chain output {
type filter hook output priority 0; policy drop;
ct state related,established accept
oif lo accept
udp dport { 53, 123 } accept
tcp dport { 53, 80, 443 } accept
icmp type echo-request accept
}
The containers' packets are not evaluated by these chains, but by the forward chain and won't be restricted.
IPv6
Using the inet family without enabling correctly ICMPv6 prevents any IPv6 connectivity, because IPv6 doesn't rely on (almost never-firewalled) ARP but on ICMPv6 for link local connectivity. Either use the ip family (and use an other name than filter for the table to avoid any clash with iptables-nft) or deal correctly with ICMPv6: accept them all or check which are required in input and in output direction for correct SLAAC (NDP: RS, RA, NS, NA, ...), ping ... handling.
