Why is the Linux command `unshare --pid=p --mount=m` not creating a persistent namespace?

Question

From everything I have read in the unshare and nsenter man pages, I should be able to bind-mount a directory to itself, mount --make-private the directory, and then use files within that directory to hold refs for persistent namespaces. Here is what I'm doing, basically the same as the man unshare but with different directories and using --pid=file in addition to --mount=file

Terminal 1:

# mkdir -p /mnt/jails/debian/bookworm/.ns
# mount --bind /mnt/jails/debian/bookworm/.ns /mnt/jails/debian/bookworm/.ns
# touch /mnt/jails/debian/bookworm/.ns/{mount,pid}
# mount --make-private /mnt/jails/debian/bookworm/.ns
# unshare --fork --mount-proc --mount=/mnt/jails/debian/bookworm/.ns/mount --pid=/mnt/jails/debian/bookworm/.ns/pid /bin/sh & echo $!; fg
[1] 151299
151299
sh-4.4# echo $$
1
sh-4.4# grep NS /proc/self/status
NStgid: 3
NSpid:  3
NSpgid: 3
NSsid:  0

So far so good, the container above is working. While that runs:

Terminal 2:

# nsenter  --mount=/mnt/jails/debian/bookworm/.ns/mount --pid=/mnt/jails/debian/bookworm/.ns/pid /bin/sh
sh-4.4# ps ax
Error, do this: mount -t proc proc /proc
# ls /proc/1/exe -l
lrwxrwxrwx. 1 root root 0 Jul 21 18:49 /proc/1/exe -> /usr/bin/bash
sh-4.4# mount -t proc proc /proc
sh-4.4# ps ax|head
<shows pids from the host OS, not from the container>
sh-4.4# grep NS /proc/self/status
NStgid: 156987
NSpid:  156987
NSpgid: 156987
NSsid:  156921

I've also tried this in Terminal 2 (note the pid from Terminal 1) with the exact same results:

# nsenter -t 151299 -a  /bin/sh
sh-4.4# ps ax
Error, do this: mount -t proc proc /proc
# ls /proc/1/exe -l
lrwxrwxrwx. 1 root root 0 Jul 21 18:49 /proc/1/exe -> /usr/bin/bash
sh-4.4# mount -t proc proc /proc
sh-4.4# ps ax|head
<shows pids from the host OS, not from the container>
sh-4.4# grep NS /proc/self/status
NStgid: 155356
NSpid:  155356
NSpgid: 155356
NSsid:  143538

For some reason nsenter is entering the host OS's pid space, however it does seem to see a the namespace of the correct /proc directory, but it is invalid for sh in terminal2 because the pid namespace isn't working so (I think) thats why ps ax gives an error. Also I've tried both with and without --mount-proc

Questions:

How can I enter the PID namespace from Terminal 1?

What am I doing wrong here?

(Host linux kernel is 5.18 running Oracle Linux 8.)

Answer 1

There is a bug between before util-linux v2.36 that was patched in this commit:

0d5260b66 unshare: Fix PID and TIME namespace persistence

Use a version of util-linux containing the patch!

Here is a test-script to verify if you have this bug:

umount -l /mnt/jails/*/*/.ns/* /mnt/jails/*/*/.ns/
sleep 1

mkdir -p /mnt/jails/debian/bookworm/.ns
mount --bind /mnt/jails/debian/bookworm/.ns /mnt/jails/debian/bookworm/.ns
touch /mnt/jails/debian/bookworm/.ns/{mount,pid}
mount --make-private /mnt/jails/debian/bookworm/.ns

unshare --fork --mount-proc --mount=/mnt/jails/debian/bookworm/.ns/mount --pid=/mnt/jails/debian/bookworm/.ns/pid sleep 99 &
upid=$!

sleep 1

if nsenter --mount=/mnt/jails/debian/bookworm/.ns/mount --pid=/mnt/jails/debian/bookworm/.ns/pid [ -d /proc/self ]; then
    kill $upid
    echo worked
    exit 0
else
    kill $upid
    echo didnt
    exit 1
fi

Answer 2

First, I am suspicious of this business:

...  /bin/sh & echo $!; fg

That seems to always end up with the shell exiting and my terminal in a weird state. So don't do that; just run:

unshare --fork --mount-proc --mount=/mnt/jails/debian/bookworm/.ns/mount --pid=/mnt/jails/debian/bookworm/.ns/pid

Now you have a shell running in its own mount and pid namespaces.

---

I've also tried this in Terminal 2 (note the pid from Terminal 1) with the exact same results:

We discussed this elsewhere; your life will be simpler if you stop mucking about with process ids, especially since you've already gone to the effort of creating persistent namespaces. Intead of -t <pid>, use those persistent namespaces you created:

# nsenter --mount=/mnt/jails/debian/bookworm/.ns/mount --pid=/mnt/jails/debian/bookworm/.ns/pid
# ps -fe
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 16:45 pts/4    00:00:00 /bin/sh
root           3       0  1 16:48 pts/15   00:00:00 -bash
root          41       3  0 16:48 pts/15   00:00:00 ps -fe

As you can see, we've successfully entered the namespaces created by the earlier unshare command.