Ubuntu server, packets from outside network not reaching local socket but seen on tcpdump

Question

I am running Ubuntu server 22.04, and have a peculiar port forwarding issue with a local machine. This machine has two Ethernet interfaces and the connected enp1s0 interface has an IP address 192.168.1.50. Full ip configuration is following:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:e0:4f:68:02:bb brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.50/24 metric 100 brd 192.168.1.255 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet6 240f:74:de92::d1e/128 scope global dynamic noprefixroute 
       valid_lft 35597sec preferred_lft 35597sec
    inet6 fd41:d8b6:99ba::d1e/128 scope global dynamic noprefixroute 
       valid_lft 35597sec preferred_lft 35597sec
    inet6 fd41:d8b6:99ba:0:2e0:4fff:fe68:2bb/64 scope global mngtmpaddr noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 240f:74:de92:0:2e0:4fff:fe68:2bb/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 98462sec preferred_lft 98462sec
    inet6 fe80::2e0:4fff:fe68:2bb/64 scope link 
       valid_lft forever preferred_lft forever
3: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 00:e0:4c:68:01:6e brd ff:ff:ff:ff:ff:ff
    altname enp2s0
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:55:86:e6:e5 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

I have set up port forwarding rules for my router so that any TCP or UDP packets coming from wan to port 22211 are forwarded to lan 192.168.1.50 port 22211

In my ufw configuration, I have allowed allowed routing of this port:

$ sudo ufw status
[sudo] password for *
Status: active

To                         Action      From
--                         ------      ----
22211/tcp                  ALLOW       Anywhere                  
22211/tcp (v6)             ALLOW       Anywhere (v6)             

Now, if I start a simple netcat socket (nc -l -p 22211) for this port, I can reach it with telnet via another machine in the local network just fine. Here is tcudump log when I telnet from another local machine. I connect, and send a letter a and press enter:

$ sudo tcpdump -pnvvi enp1s0 port 22211
tcpdump: listening on enp1s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
21:01:48.350024 IP (tos 0x0, ttl 64, id 59572, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.1.196.38840 > 192.168.1.50.22211: Flags [S], cksum 0x527d (correct), seq 3440167578, win 32120, options [mss 1460,sackOK,TS val 3772353734 ecr 0,nop,wscale 7], length 0
21:01:48.350139 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.1.50.22211 > 192.168.1.196.38840: Flags [S.], cksum 0x3a60 (correct), seq 495600843, ack 3440167579, win 65160, options [mss 1460,sackOK,TS val 102903428 ecr 3772353734,nop,wscale 7], length 0
21:01:48.351892 IP (tos 0x0, ttl 64, id 59573, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.1.196.38840 > 192.168.1.50.22211: Flags [.], cksum 0x66b7 (correct), seq 1, ack 1, win 251, options [nop,nop,TS val 3772353737 ecr 102903428], length 0
21:01:50.698846 IP (tos 0x0, ttl 64, id 59574, offset 0, flags [DF], proto TCP (6), length 55)
    192.168.1.196.38840 > 192.168.1.50.22211: Flags [P.], cksum 0xf275 (correct), seq 1:4, ack 1, win 251, options [nop,nop,TS val 3772356082 ecr 102903428], length 3

But when I try telnet form outside my local network, for some reason the connection is never established. For the longest time I was sure that my port forwarding configuration was off (although same router forward other ports to another machine in my local network just fine, and I basically just mirrored the configuration to another port), but as you can see from tcpdump logs below, packets arrive to the enp1s0 interface:

$ sudo tcpdump -pnvvi enp1s0 port 22211
tcpdump: listening on enp1s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
20:58:05.448829 IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    126.33.109.168.9875 > 192.168.1.50.22211: Flags [S], cksum 0x4448 (correct), seq 2086171535, win 65535, options [mss 1240,nop,wscale 6,nop,nop,TS val 560510789 ecr 0,sackOK,eol], length 0
20:58:06.593532 IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    126.33.109.168.9875 > 192.168.1.50.22211: Flags [S], cksum 0x405f (correct), seq 2086171535, win 65535, options [mss 1240,nop,wscale 6,nop,nop,TS val 560511790 ecr 0,sackOK,eol], length 0
20:58:07.613818 IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    126.33.109.168.9875 > 192.168.1.50.22211: Flags [S], cksum 0x3c76 (correct), seq 2086171535, win 65535, options [mss 1240,nop,wscale 6,nop,nop,TS val 560512791 ecr 0,sackOK,eol], length 0
20:58:08.603603 IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    126.33.109.168.9875 > 192.168.1.50.22211: Flags [S], cksum 0x388c (correct), seq 2086171535, win 65535, options [mss 1240,nop,wscale 6,nop,nop,TS val 560513793 ecr 0,sackOK,eol], length 0
20:58:09.563590 IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    126.33.109.168.36371 > 192.168.1.50.22211: Flags [S], cksum 0xcd21 (correct), seq 2086171535, win 65535, options [mss 1240,nop,wscale 6,nop,nop,TS val 560514795 ecr 0,sackOK,eol], length 0

I can also see that netcat is listening to all interfaces (the 0.0.0.0 part):

$ sudo ss -tulpn | grep 22211
tcp   LISTEN 0      1                               0.0.0.0:22211      0.0.0.0:*    users:(("nc",pid=3344,fd=3))

The routing table looks like following:

$ sudo route -vn
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.254   0.0.0.0         UG    0      0        0 enp1s0
0.0.0.0         192.168.1.1     0.0.0.0         UG    100    0        0 enp1s0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
192.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 enp1s0
192.168.1.1     0.0.0.0         255.255.255.255 UH    100    0        0 enp1s0

So I am lost why the packets coming from outside network never reach the local socket in port 22211... Is there some additional firewall configuration that I'm supposed to configure ?

Edit: interestingly, indeed ping 8.8.8.8 times out (while for example ping google.com works fine)

Answer 1

TL;DR

If something complicated doesn't work, then make sure the basics are functioning correctly first.

Longer version

Preamble

This is a good example of how to ask a question.

1. There is background information, e.g. it is running Ubuntu 22.04, the machine has two ethernet interfaces.

2. Detailed information, e.g. the ip config information is provided.

3. Both working and non-working examples are shown.

Debugging the issue.

Correct misunderstandings.

• The OP stated "So I am lost why the packets coming from outside network never reach the local socket in port 22211." when the data showed that the packets were arriving, it was the responses which were not going out of that interface.

• The OP said that it had two interfaces. The ip information shows that the second interface is down, but maybe the machine was expecting to send the response out of the other interface?

First request for more information.

• "Does the routing table show that the route to 126.33.189.168 is via the interface that is up (enp1s0)?" The OP responded by adding in the routing table and also said

• "I added the routing table information to the original message. Also I made sure that the other interface eno1 is down, but there appears to be no change"

Second request for more information.

The routing table had symbolic names in it, rather than numeric addresses. Without knowing the mapping this wasn't too useful, so

• "Could you change your edit to show the output of sudo route -vn to show the numeric values please?"

Ask things in a way that is natural for the responder.

A comment suggested using ip route instead of route -vn. I have used ip route for years, and would assert that it is a better tool. The question to ask is "are we trying to teach the OP better tools or are we trying to help him fix his issue?". I think it would be a distraction to start introducing new tools.

Second request for more information - continued.

• "Can you also confirm that you can ping some well known address from 192.168.1.50, e.g. 8.8.8.8?" This elicited the response

• 'You are right, for some reason ping 8.8.8.8 times out! But ping google.com works fine."

Debug why ping google.com works

• "And what IP address does ping google.com choose? (It's shown in the first line.) This sounds like you've firewalled yourself away from IPv4, but still have fully functioning IPv6. "

I wouldn't use the term firewalled. With the routing table showing 2 default routes with different metrics this was now almost certainly a routing issue.

Third request for more information.

Most people only have a single connection to the internet, but try not to make any unwarrented assumptions.

• "Do you expect to have 2 routers each of which can reach the internet (perhaps with 2 different ISPs)?"

when the answer came back negative, it was just a matter of telling the OP to fix the routing issue (using the command they were familier with) and all is working. Anticipating the answer and putting the instructions in with the question saves delay.

• "Being able to ping 8.8.8.8 is your most important issue. Resolving that may well resolve everything."

The actual problem.

• when packets need to go out to the internet, rather than the local network, they were told to go via a non existant machine.

• In order to send packets to this non existent machine the host was sending ARP packets, which were not getting a response.

• eventually the host gave up trying to contact the non existent machine and reported an error to the ping command, or didn't send an acknowlegment to the incoming connect.

• fixing the routing table to send packets destined to the internet via the correct address made everything work.