Mounting a NFS Directory into Host Volume that is shared with Docker

Question

Consider the following Docker container:

docker run --rm -it -v /tmp:/mnt/tmp alpine sh

This mounts the host directory /tmp into /mnt/tmp inside of the alpine container.

Now, on the Host System I mount a NFS volume to the /tmp directory:

mkdir /tmp/nfs
mount -t nfs4 192.168.1.100:/data /tmp/nfs

The mount works on the Host System, and I see the following:

# ls /tmp/nfs
file1 file2 file3
#

But on the Docker Container, I see a blank directory:

# ls /mnt/tmp/nfs
#

I know that I can get around this by doing the mount directly in the Docker Container. But I am really interested to know why the mount works on the host container but not in the docker container?

Answer 1

This happens because the volume is using private mount propagation. This means that once the mount happens, any changes that happen on the origin side (e.g. the "host" side in the case of Docker) will not be visible underneath the mount.

There are a couple of ways to handle this:

1. Do the NFS mount first, then start the container. The mount will propagate to the container, however as before any changes to the mount will not be seen by the container (including unmounts).

2. Use "slave" propagation. This means that once the mount is created, any changes on the origin side (docker host) will be able to be seen in the target (in the container). If you happen to be doing nested mounts, you'll want to use rslave (r for recursive).

There is also "shared" propagation. This mode would make changes to the mountpoint from inside the container propagate to the host, as well as the other way around. Since your user wouldn't even have privileges to make such changes (unless you add CAP_SYS_ADMIN), this is probably not what you want.

You can set the propagation mode when creating the mount like so:

$ docker run -v /foo:/bar:private

The other alternative would be to use a volume rather than a host mount. You can do this like so:

$ docker volume create \
    --name mynfs \
    --opt type=nfs \
    --opt device=:<nfs export path> \
    --opt o=addr=<nfs host> \
    mynfs
$ docker run -it -v mynfs:/foo alpine sh

This will make sure to always mount in the container for you, doesn't rely on having the host setup in some specific way or dealing with mount propagation.
note: the : at the front of the device path is required, just something weird about the nfs kernel module.
note: Docker does not currently resolve <nfs host> from a DNS name (it will in 1.13) so you will need to supply the ip address here.

More details on "shared subtree" mounts: https://www.kernel.org/doc/Documentation/filesystems/sharedsubtree.txt

Answer 2

Enable shared mount propagation on the volume by adding the :shared flag at the end of the volume argument:

docker run --rm -it -v /tmp:/mnt/tmp:shared alpine sh

If Docker was installed through a package manager or install script for systemd, you may need to adjust the MountFlags daemon argument. To do that, locate the docker.service file:

$ sudo find /etc -name "docker.service"

In my case on Ubuntu 16.04, it was located at /etc/systemd/system/multi-user.target.wants/docker.service. Edit this file with vi or nano, and ensure that the MountFlags option reads:

MountFlags=shared

Save the file, reload the daemon args, and restart docker:

$ sudo systemctl daemon-reload
$ sudo systemctl restart docker

Now you should be able to set the shared mount propagation flag on volumes when using "docker run".

Answer 3

Starting from docker 17.06, you can mount NFS shares to the container directly when you run it, without the need of extra capabilities

export NFS_VOL_NAME=mynfs NFS_LOCAL_MNT=/mnt/mynfs NFS_SERVER=my.nfs.server.com NFS_SHARE=/my/server/path NFS_OPTS=vers=4,soft

docker run --mount \
  "src=$NFS_VOL_NAME,dst=$NFS_LOCAL_MNT,volume-opt=device=:$NFS_SHARE,\"volume-opt=o=addr=$NFS_SERVER,$NFS_OPTS\",type=volume,volume-driver=local,volume-opt=type=nfs" \
  busybox ls $NFS_LOCAL_MNT

Alternatively, you can create the volume before the container:

docker volume create --driver local \
  --opt type=nfs --opt o=addr=$NFS_SERVER,$NFS_OPTS \
  --opt device=:$NFS_SHARE $NFS_VOL_NAME

docker run --rm -v $NFS_VOL_NAME:$NFS_LOCAL_MNT busybox ls $NFS_LOCAL_MNT

Got the hint from https://github.com/moby/moby/issues/28809