Create dynamic blacklist with nftables

Question

I want to create a dynamic blacklist with nftables. Under version 0.8.3 on the embedded device I create a ruleset looks like this with nft list ruleset:

table inet filter {
set blackhole {
    type ipv4_addr
    size 65536
    flags timeout
}

chain input {
    type filter hook input priority 0; policy drop;
    ct state invalid drop
    ct state established,related accept
    iif "lo" accept
    ip6 nexthdr 58 icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, echo-request, echo-reply, mld-listener-query, mld-listener-report, mld-listener-done, nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, ind-neighbor-solicit, ind-neighbor-advert, mld2-listener-report } accept
    ip protocol icmp icmp type { echo-reply, destination-unreachable, echo-request, router-advertisement, router-solicitation, time-exceeded, parameter-problem } accept
    ip saddr @blackhole counter packets 0 bytes 0 drop
    tcp flags syn tcp dport ssh meter flood { ip saddr timeout 1m limit rate over 10/second burst 5 packets}  set add ip saddr timeout 1m @blackhole drop
    tcp dport ssh accept
}

chain forward {
    type filter hook forward priority 0; policy drop;
}

chain output {
    type filter hook output priority 0; policy accept;
}
}

For me this is only a temporary solution. I want to use the example from the official manpage for dynamic blacklisting. If I use the offical example from the manpage my nftables file looks like this:

table inet filter {
set blackhole{
        type ipv4_addr
        flags timeout
        size 65536
}
chain input {
        type filter hook input priority 0; policy drop;

        # drop invalid connections
        ct state invalid drop

        # accept traffic originating from us
        ct state established,related accept

        # accept any localhost traffic
        iif lo accept

        # accept ICMP
        ip6 nexthdr 58 icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, echo-request, echo-reply, mld-listener-query, mld-listener-report, mld-listener-done, nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, ind-neighbor-solicit, ind-neighbor-advert, mld2-listener-report } accept
        ip protocol icmp icmp type { destination-unreachable, router-solicitation, router-advertisement, time-exceeded, parameter-problem, echo-request, echo-reply } accept

        # accept SSH (port 22)
        ip saddr @blackhole counter drop
        tcp flags syn tcp dport ssh meter flood { ip saddr timeout 10s limit rate over 10/second} add @blackhole { ip saddr timeout 1m } drop
        tcp dport 22 accept

}


chain forward {
        type filter hook forward priority 0; policy drop;
}

chain output {
        type filter hook output priority 0; policy accept;
}

}

But when I load this nftables file on version 0.8.3 with nft -f myfile I get this error:

Error: syntax error, unexpected add, expecting newline or semicolon
    tcp flags syn tcp dport ssh meter flood { ip saddr timeout 10s limit rate over 10/second} add @blackhole { ip saddr timeout 1m } drop

I don't know why this is the case, but according to the wiki it should work from version 0.8.1 and kernel 4.3.

I have version 0.8.3 and kernel 4.19.94.

I have tested under Debian Buster the ruleset from the official manpage with version 0.9.0. The ruleset from the manpage works fine with Debian, but the ip is blocked only once.

With this example I want to create a firewall rule which blocks the ip adress on ssh port if an brute force attack is started to my device. But I want to block the ip e.g for 5 minutes. After that time it should be possible to connect again to the device from the attackers ip. If he do brute force again it should be block the ip again for 5 minutes and so on. I want to avoid to use another software for my embedded device like sshguard or fail2ban if it is possible with nftables.

I hope anyone can help me. Thanks!

Answer 1

The hydra tool connects concurrently multiple times to the SSH server. In OP's case (comment: hydra -l <username> -P </path/to/passwordlist.txt> -I -t 6 ssh://<ip-address>) it will use 6 concurrent threads connecting.

Depending on server settings, one connection could typically try 5 or 6 passwords and taking about 10 seconds before being rejected by the SSH server, so I fail to see how a rate of 10 connection attempts per second could be exceeded (but that's the case). It could mean that what triggers is that more than 5 connection attempts are done in less than 1/2s. I wouldn't trust too much the accuracy of 10/s, but it can be assumed it happens here.

Version and syntax issues

The syntax not working with versions 0.8.1 or 0.8.3 is a newer syntax that appeared in this commit:

src: revisit syntax to update sets and maps from packet path

For sets, we allow this:

  nft add rule x y ip protocol tcp update @y { ip saddr}

[...]

It was committed after version 0.8.3 so available only with nftables >= 0.8.4

The current wiki revision for Updating sets from the packet path, in the same page, still displays commands with the former syntax

 % nft add rule filter input set add ip saddr @myset

[...]

and results displayed with the newer syntax:

[...]

                add @myset { ip saddr }

[...]

Some wiki pages or the latest manpage might not work with older nftables versions.

Anyway, if running with kernel 4.19, nftables >= 0.9.0 should be preferred to get additional features. For example it's available in Debian 10 or in Debian 9 backports.

Blacklisting should be done before stateful accept rules

Once the IP is added to the blacklist, this doesn't prevent established connections to continue, unhindered and unaccounted, until they're disconnected by the SSH server itself. That's because there's the usual existing short-circuit rule before:

        # accept traffic originating from us
        ct state established,related accept

This comment is misleading: it doesn't accept traffic originating from us but any traffic already ongoing. This is a short-circuit rule. Its role is to handle stateful connections by parsing all rules only for new connections: any rule after this one applies to new connections. Once connections are accepted, their individual packets stay accepted until end of connection.

For the specific case of blacklist handling, specific blacklist rules or part of them should be placed before this short-circuit rule to be able to take effect immediately. In OP's case that is:

        ip saddr @blackhole counter drop

It should be moved before the ct state established,related accept rule.

Now once the attacker is added to the blacklist, other ongoing connections won't get some remaining free attempts at guessing a password: they'll immediately hang.

If there's a blacklist, consider a whitelist

As a side note, the cheap iif lo accept rule could itself be moved before both as an optimization and to be whitelisted: all (even long lived) local established connection will also now be subject to blacklisting in case of abuse (eg: from 127.0.0.1). Consider adding various whitelisting rules before the @blackhole rule.

Optionally warn applications faster

To also prevent ongoing replies from server to reach the blacklisted IP (especially for UDP traffic, not that useful for TCP, including SSH), the equivalent rule using daddr can also be added in the inet filter output chain, with reject to inform faster the local processes trying to emit that they should abort:

    ip daddr @blackhole counter reject

Difference between add and update applied on a set

Now with such settings in place, even if ongoing connections are immediately stopped, the attacker is able to keep trying and get a new short window 1mn later, which is not optimal.

The entries must be updated in the input @blackhole ... drop rule. update will refresh the timer if the entry already existed, while add would do nothing. This will keep blocking any further (unsuccessful) attempt to connect to the SSH server until attacker gives up, with zero opened window. (The output rule I added above shouldn't be changed, it's not the attacker's actions):

replace:

ip saddr @blackhole counter drop

with (still keeping older syntax):

ip saddr @blackhole counter set update ip saddr timeout 1m @blackhole drop

It should even be moved before the ct state invalid rule, else if attacker tries invalid packets (eg TCP packet not part of a known connection, like a late RST from an already forgotten connection), the set won't be updated while it could have been.

Limit the maximum number of established connections

Requires kernel >= 4.18 and nftables >= 0.9.0, so can't be done with OP's current configuration.

The attacker might discover it can't connect too many times at once but can still keep adding, without limit, new connections, as long as not connecting too fast.

A limit on concurrent connections (as available with iptables's connlimit) can also be added with an other meter rule:

tcp flags syn tcp dport 22 meter toomanyestablished { ip saddr ct count over 3 } reject with tcp reset

will allow any given IP address to have only 3 established SSH connections.

Or while at it, instead, also trigger the @blackhole set (using newer syntax this time):

tcp flags syn tcp dport 22 meter toomanyestablished { ip saddr ct count over 3 } add @blackhole { ip saddr timeout 1m } drop

This should trigger even before the previous meter rule in OP's case. Use with care to avoid legitimate users to be affected (but see openssh's ControlMaster option).

IPv4 and IPv6

As there's no generic IPv4+IPv6 set address type, all rules handling IPv4 (whenever there's the 2-letters word ip) should probably be duplicated into a mirror rule having ip6 in them and working on an IPv6 set.