Why does bind mounting a file after unlink fail with ENOENT?

Question

I do not understand why I get ENOENT when bind-mounting after unlink:

kduda@penguin:/tmp$ echo hello > a
kduda@penguin:/tmp$ touch b c
kduda@penguin:/tmp$ sudo unshare -m
root@penguin:/tmp# mount -B a b
root@penguin:/tmp# rm a
root@penguin:/tmp# cat b
hello
root@penguin:/tmp# mount -B b c
mount: mount(2) failed: No such file or directory

This seems like a bug to me. You can even re-create "a", pointing to the same exact inode, but you get the same thing:

kduda@penguin:/tmp$ echo hello > a
kduda@penguin:/tmp$ ln a a-save
kduda@penguin:/tmp$ sudo unshare -m
root@penguin:/tmp# mount -B a b
root@penguin:/tmp# rm a
root@penguin:/tmp# ln a-save a
root@penguin:/tmp# mount -B b c
mount: mount(2) failed: No such file or directory

What in the world is going on?

Answer 1

The mount(2) system call will completely resolve its paths through mounts and symlinks, but unlike open(2), will not accept a path to a deleted file, ie a path which resolves to a unlinked directory entry.

(similar to the <filename> (deleted) paths of /proc/PID/fd/FD, procfs will display unlinked dentries as <filename>//deleted in /proc/PID/mountinfo)

# unshare -m
# echo foo > foo; touch bar baz quux
# mount -B foo bar
# mount -B bar baz
# grep foo /proc/self/mountinfo
56 38 8:7 /tmp/foo /tmp/bar ...
57 38 8:7 /tmp/foo /tmp/baz ...

# rm foo
# grep foo /proc/self/mountinfo
56 38 8:7 /tmp/foo//deleted /tmp/bar ...
57 38 8:7 /tmp/foo//deleted /tmp/baz ...
# mount -B baz quux
mount: mount(2) failed: /tmp/quux: No such file or directory

All this used to work in older kernels, but does not since v4.19, first introduced by this change:

commit 1064f874abc0d05eeed8993815f584d847b72486
Author: Eric W. Biederman <[email protected]>
Date:   Fri Jan 20 18:28:35 2017 +1300

    mnt: Tuck mounts under others instead of creating shadow/side mounts.
...
+       /* Preallocate a mountpoint in case the new mounts need
+        * to be tucked under other mounts.
+        */
+       smp = get_mountpoint(source_mnt->mnt.mnt_root);
+       if (IS_ERR(smp))
+               return PTR_ERR(smp);
+

It looks that this effect was unintended by the change. Since then other unrelated changes have piled on, confusing it even more.

A consequence of it is that it also prevents pinning a deleted file somewhere else in the namespace via an open fd to it:

# exec 7>foo; touch bar
# rm foo
# mount -B /proc/self/fd/7 bar
mount: mount(2) failed: /tmp/bar: No such file or directory

The last command fails because of the same condition as the OP's.

You can even re-create a, pointing to the same exact inode, but you get the same thing

It's the same thing as with /proc/PID/fd/FD "symlinks". The kernel is smart enough to follow a file through straight renames, but not through ln + rm (link(2) + unlink(2)):

# unshare -m
# echo foo > foo; touch bar baz
# mount -B foo bar
# mount -B bar baz
# grep foo /proc/self/mountinfo
56 38 8:7 /tmp/foo /tmp/bar ...
57 38 8:7 /tmp/foo /tmp/baz ...

# mv foo quux
# grep bar /proc/self/mountinfo
56 38 8:7 /tmp/quux /tmp/bar ...

# ln quux foo; rm quux
# grep bar /proc/self/mountinfo
56 38 8:7 /tmp/quux//deleted /tmp/bar ...

Answer 2

Walking through the source code, I found exactly one ENOENT that was relevant, i.e. for an unlinked directory entry:

static int attach_recursive_mnt(struct mount *source_mnt,
            struct mount *dest_mnt,
            struct mountpoint *dest_mp,
            struct path *parent_path)
{
    [...]

    /* Preallocate a mountpoint in case the new mounts need
     * to be tucked under other mounts.
     */
    smp = get_mountpoint(source_mnt->mnt.mnt_root);

static struct mountpoint *get_mountpoint(struct dentry *dentry)
{
    struct mountpoint *mp, *new = NULL;
    int ret;

    if (d_mountpoint(dentry)) {
        /* might be worth a WARN_ON() */
        if (d_unlinked(dentry))
            return ERR_PTR(-ENOENT);

https://elixir.bootlin.com/linux/v5.2/source/fs/namespace.c#L3100

get_mountpoint() is generally applied to the target, not the source. In this function, it is called because of mount propagation. It is necessary to enforce the rule that you cannot add mounts on top of a deleted file, during mount propagation. But the enforcement is happening eagerly, even if no mount propagation happens that would require this. I think it is good that the checking is consistent like this, it is just coded a bit more obscurely than I would ideally prefer.

Either way I look at it, I think it is reasonable to enforce this. So long as it helps reduce the number of weird cases to analyze, and no-one has an especially compelling counter-argument.

Answer 3

A comprehensive answer is: there are three things to understand, and then this all makes sense.

First, the source of a bind-mount is a dentry, not an inode. That is, you don't bind-mount an inode on a name; you bind-mount one dentry over another dentry. To see the difference, look at what happens if you mount different links to the same inode; the mounts are different, because the source dentries are different, even when the inode is the same:

root@penguin:/tmp# echo hello > a1
root@penguin:/tmp# ln a1 a2
root@penguin:/tmp# touch b1 b2
root@penguin:/tmp# mount -B a1 b1
root@penguin:/tmp# mount -B a2 b2
root@penguin:/tmp# ls -li a1 a2 b1 b2
9552271 -rw-r--r-- 2 root root 6 Aug 25 05:16 a1
9552271 -rw-r--r-- 2 root root 6 Aug 25 05:16 a2
9552271 -rw-r--r-- 2 root root 6 Aug 25 05:16 b1
9552271 -rw-r--r-- 2 root root 6 Aug 25 05:16 b2
root@penguin:/tmp# grep /tmp/ /proc/self/mountinfo
421 364 0:38 /lxd/.../rootfs/tmp/a1 /tmp/b1 rw,...
422 364 0:38 /lxd/.../rootfs/tmp/a2 /tmp/b2 rw,...

The second thing to understand is that when you mount something that is itself the target of an earlier bind-mount, that's the same dentry object as the source of the bind-mount (that's what bind-mounting is; one dentry over another.) So, if a1 is mounted on b1, then mounting b1 on c1 is exactly the same thing as mounting a1 on c1 because the names a1 and b1 refer to the same dentry.

The third thing to understand is that the kernel prohibits bind-mounting a deleted dentry because... of no good reason that I can see. It appears that an error check intended for the target of a mount (preventing mounting over a deleted dentry, which would make no sense to do, because you could never reference your new mount) is applying for no good reason to the source of the mount as well. That's this code here:

static struct mountpoint *get_mountpoint(struct dentry *dentry)
{
    struct mountpoint *mp, *new = NULL;
    int ret;

    if (d_mountpoint(dentry)) {
        /* might be worth a WARN_ON() */
        if (d_unlinked(dentry))
            return ERR_PTR(-ENOENT);

The consequence of these three facts is (continuing the shell session above) is ENOENT mounting b2 on c2 if a2 is deleted:

root@penguin:/tmp# touch c1 c2
root@penguin:/tmp# rm a2
root@penguin:/tmp# mount -B b1 c1
root@penguin:/tmp# mount -B b2 c2
mount: mount(2) failed: /tmp/c2: No such file or directory
root@penguin:/tmp# 

This strikes me as a bug because your b2-on-c2 mount is valid if you delete a2 after the mount, and the order should not matter: either having a deleted dentry mounted on something is legal or it isn't, it should not matter when it became deleted. However, reasonable people disagree.

Thanks everyone.