mounting /var as overlayfs

Question

I'm running a read only filesystem on a raspberry pi so far everything works fine until i tried to mount /var as overlayfs for nginx and other services to work using this:

VAROVRL="-o lowerdir=/var,upperdir=/mnt/persist/var-rw,workdir=/mnt/persist/var-work"
mount -t overlay ${VAROVRL} overlay /var

while this is working and all services start no issues i noticed that the mount command outputs only the overlay mount and it gets duplicated every time i reboot.

after 3 reboots:

mount
overlay on /var type overlay (rw,lowerdir=/var,upperdir=/mnt/persist/var-rw,workdir=/mnt/persist/var-work)
overlay on /var type overlay (rw,lowerdir=/var,upperdir=/mnt/persist/var-rw,workdir=/mnt/persist/var-work)
overlay on /var type overlay (rw,lowerdir=/var,upperdir=/mnt/persist/var-rw,workdir=/mnt/persist/var-work)

output of /etc/mount

/dev/root / ext4 ro,relatime,data=ordered 0 0
devtmpfs /dev devtmpfs rw,relatime,size=469532k,nr_inodes=117383,mode=755 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/net_cls cgroup rw,nosuid,nodev,noexec,relatime,net_cls 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=23,pgrp=1,timeout=300,minproto=5,maxproto=5,direct 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
tmpfs /tmp tmpfs rw,relatime,size=102400k 0 0
/dev/mmcblk0p1 /boot vfat ro,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro 0 0
/dev/mmcblk0p5 /mnt/persist ext4 rw,relatime,data=ordered 0 0
/dev/mmcblk0p6 /mnt/cache ext4 rw,relatime,data=ordered 0 0
/dev/mmcblk0p7 /mnt/osboot vfat rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro 0 0
/dev/mmcblk0p8 /mnt/osimage ext4 rw,relatime,data=ordered 0 0
/dev/mmcblk0p9 /mnt/userdata ext4 rw,relatime,data=ordered 0 0
overlay /etc overlay rw,relatime,lowerdir=/etc,upperdir=/mnt/persist/etc-rw,workdir=/mnt/persist/etc-work 0 0
overlay /var overlay rw,relatime,lowerdir=/var,upperdir=/mnt/persist/var-rw,workdir=/mnt/persist/var-work 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0

output of /etc/mtab

overlay /var overlay rw,lowerdir=/var,upperdir=/mnt/persist/var-rw,workdir=/mnt/persist/var-work 0 0
overlay /var overlay rw,lowerdir=/var,upperdir=/mnt/persist/var-rw,workdir=/mnt/persist/var-work 0 0
overlay /var overlay rw,lowerdir=/var,upperdir=/mnt/persist/var-rw,workdir=/mnt/persist/var-work 0 0

note that /etc is also mounted as overlayfs but does not generate this problem when it's the only overlay mount.

anyone can spot something i'm doing wrong here?

Answer 1

The file /etc/mtab is written by the mount and umount commands. Keeping it accurate requires a bit of work because they can only update /etc/mtab if that file is available and writable.

For the usual case where /etc is mounted read-write at some point during boot, distributions set up a script that rewrite /etc/mtab during startup, as soon as the root partition has been mounted read-write. This is necessary in case the system shut down without unmounting everything (e.g. due to a system crash or power failure).

In your case, where /etc is on overlayfs, either the startup script runs at the wrong time when /etc is still read-only, or it doesn't support the case of an overlay root. So if you want to keep /etc/mtab as a regular file, you'll have to tweak this script or the time when it's executed.

But you probably don't need to do this. A common setup is to have /etc/mtab be a symbolic link to /proc/mounts. The two files contain mostly the same information with mostly the same syntax; from the point of view of applications that read them, they're compatible. Since /proc/mounts reflects the current kernel information, it is always up-to-date, and the mount and umount commands won't touch them.

The downside of /proc/mounts compared with /etc/mtab is that it shows information (especially mount options) as printed back by the kernel, rather than the exact parameters passed to the mount command. So a little information is lost. That information is rarely useful though.