Application server socket won't open in docker on my machine

Question

Introduction

• I am developing a proprietary application that uses libevent with libevhtp to create a server socket for http communication.
• I wrote a Dockerfile to place this application in a docker image.

The resulting docker image, that starts the application per default, runs fine on other computers that I have tested (ubuntu 20.04/22.04) by running the image via
docker run --rm -p 8080:8080 -t mytag

• I can connect to port 8080 of that computer just fine from within the same network.
• I can docker exec -ti runningcontainer bash of the running container and netstat -pntl to see the application is listening on port 8080.

The Issue

On my laptop this isn't the case. I'm not running Ubuntu like the other servers I have tried it on. I'm on Manjaro. I can't see how that would matter though.

• Inside the running docker container on my laptop with the running application, netstat -pntl shows no listening socket
• It follows that I can't connect to port 8080 of the application - curl localhost:8080 returns Connection Refused
• Running the application on my computer natively (not in docker) works as expected, I can access port 8080
• Running nc -l 0.0.0.0 9999 inside my docker container that is running on my laptop opens a socket and netstat reports it listening.
• The connection via nc 127.0.0.1 9999 from inside the container to that open socket works as expected.
• I have tried running the container with --privileged with the same result - not working
• I have pruned all docker images, containers, networks, uninstalled and removed all traces of docker from my computer and reinstalled it without any change in behavior

Conclusion

It seems to me that the application is not opening any listening sockets when running inside the docker container for some reason. The logs of the implementation show no errors - Note: I have not checked the implementation for errors or some overlooked error code.

I need help figuring out how I could try to debug this the most efficient way.

My next steps would be to

• Create a small binary with libevhtp that serves a test.html file and try running that in docker - and if that works fine
• Slowly disabling parts of the malfunctioning application bit by bit until I can find the culprit.

The second step would be a lot of work so I'm reaching out here in case anyone has had a similar issue or could point to some things I could try before taking on this endeavor.

Answer 1

Findings

It always failed on initialization with different modules enabled that would use up the available file descriptors. By the time the socket was to be bound, there were none left. I was stuck when trying to use lsof in docker and gave up on that earlier because there are too many possibilities of what may be the cause of it not working. (see How can I substitute lsof inside a Docker (native, not LXC-based))

Finally a google search about file descriptor limits:

# uname -r
5.15.106-1-MANJARO
# ulimit -n
1024

And a helpful answer: https://stackoverflow.com/questions/24318543/ solved the issue for me.

Solution:

run docker with --ulimit parameter
docker run --rm -p 8080:8080 --ulimit nofile=262144:262144 -ti mytag:latest bash

UPDATE:

According to this thread: https://github.com/moby/moby/issues/44547
a semi-permanent solution seems to be to change LimitNOFILE value from infinity to 1048576 in file /usr/lib/systemd/system/containerd.service. You might have to repeat this after a system upgrade.
Another solution is outlined in this thread: https://bbs.archlinux.org/viewtopic.php?id=285058 where it is suggested to create a docker settings file in /etc/docker/daemon.json to set the default ulimits for docker like so:

{ 
    "default-ulimits": {
        "nofile": {
            "Hard": 1048576,
            "Name": "nofile",
            "Soft": 1048576
        }
    }
}

and then run systemctl reload docker to apply the changes.