Nftables ct count over 2 counter continues to trigger with only 1 connection

Question

I have a counter that triggers when the ct count is over 2. I am monitoring the number of connections as well as the states using ss -at and conntrack -L. The following rules do not appear to be behaving properly:

table ip mytable {
        chain mychain {
                type filter hook input priority filter; policy accept;
                ip saddr != 123.123.123.123 drop
                ip saddr 123.123.123.123 ct count over 2 counter
        }
}

To test these, I am opening TCP connections using SSH from my source address 123.123.123.123. With 1 or 2 connections nothing happens (packets 0 bytes 0) and with 3 the counter starts increasing (packets 6599 bytes 475441) as expected.

The problem is that when I then disconnect 2 of the SSH back down to 1 connection again, the counter keeps increasing. I have checked ss -at and conntrack -L and both only show 1 active connection despite the counter being triggered. There are no TIME_WAIT or other entries shown and only a single TCP connection which should not be enough to continue triggering the counter. If I fully disconnect all the connections then open a new SSH it sometimes, but not always, stops increasing again.

Why is this ct count over 2 rule behaving so strangely? It is almost as if once it triggers it struggles to deactivate regardless of how many connections are actually present.