Read/write /etc with read-only filesystem

Question

I have a setup for an embedded project where I'd like to keep the root filesystem read-only and system configuration on a read/write partition. Users cannot install programs, but they can modify configurations. Most of this is fairly easy except for the /etc directory. It contains configuration that is needed on startup (like fstab) as well as configuration that can be changed (like passwd).

I'm not sure what my best option for moving forward. Could I have a default /etc directory and mount over it after boot? How about symlink all the variable configuration files to the read/write partition? Another better solution? For simplicity, consider how this would be done using a Raspberry Pi.

Some might (rightly) ask how updates are handled. Updates are compressed and signed images of the root and boot partitions the embedded software know how to install. Hence the desire for a read-only root file system as any changes would be lost on upgrade.

Answer 1

I have two feasible options for you. The first is rather difficult. The second should be easy if you can use btrfs.

initramfs

A typical way to boot Linux today is by using an initramfs. It is basically a minimal rootfs having only what is needed to mount the real rootfs. The binaries and configurations are packed into an archive stored under /boot and on system start unpacked in a tmpfs as first rootfs.

This means you can have a boot process where a temporary rootfs is loaded from an archive you can modify where the kernel will call an executable called init which you can modify which is intended to mount the real root and doing other system startup stuff.

For a more detailed explanation of this part you can read for example the Arch boot process. (especially part 5 (initramfs) and 6 (Early userspace))

Your chance: the initramfs could be anything and do anything. The Gentoo Wiki give you a glance on what is possible.
The bad part: it is not easy. The normal way is to use some tools to create an initramfs like initramfs-tools on Debian or mkinitcpio for Arch or dracut or others. They have configuration options but having a separate /etc is to special.

For initramfs-tools (I don't know the other tools) it could be feasible to add your own mount /etc stuff. Debian typical initramfs-tools are basically a bunch of shell scripts packing a early init which is also just a shell script.

The tooling has hooks which can be used to add custom files to the initramfs (the name hook can be a bit misleading; those are run by update-initramfs when creating a new initrafs usually after a kernel update) and scripts which are executed on different points in the early userspace init (again a kinda misleading name because those are something I would call hook).

Good luck if you take this route. If you want an initramfs under Debian you can extract one (they are at /boot/initrd.image*) with the unmkinitramfs command and research how other tools integrate there, e.g. dropbear-initramfs.

Btrfs

Another much simpler option is to use btrfs as root filesystem. If using btrfs is an option for you you can utilize its subvolume feature which can be made read-only.

A btrfs subvolume can be mounted like a separate filesystem but the good part for you it does not need to. Mounting a subvolume will automatically mount all subvolume below.

You can quite simply make /etc a separate subvolume¹. After an installation you would do something like

mv /etc /etc-tmp
btrfs subvolume create /etc
mv /etc-tmp/* /etc-tmp/.* /etc
rmdir /etc-tmp

Now you have /etc in a different subvolume than everything else, you can make the root subvolume read-only with the following command and /etc remains writable.

btrfs property set -t subvol / ro true

---

¹_{By the way: the same approach can be used to make /usr read-only. Always funny if you move the existing /usr away, create the subvolume and then when you want to move the content back you are greeted with a "mv: command not found" because your /usr/bin is currently empty.}

Answer 2

I have a raspberry pi with a readonly root filesystem. There is a second partition mounted readwrite. /etc, /var, /home, and /srv are bind mounts to the readwrite partition. This will probably work with any linux.

I started with a raspberry pi image I downloaded from the internet. I wrote the raspberry pi image to a sdcard. The raspberry pi image did not fill the whole sdcard, so there was unused unformatted space at the end of the sdcard. I expanded the root partition, then used the rest of the space to create another ext4 partition. I made a few edits to /etc, like user account, hostname, timezone, etc. I moved /etc/fstab to /fstab, and made /etc/fstab a soft link to ../fstab. I copied /etc, /var, /home, and /srv to the second partition. Then I changed /fstab to:

PARTUUID=19de2757-02  /                      ext4   ro,noatime  0 1
PARTUUID=19de2757-01  /boot                  vfat   ro          0 2
PARTUUID=19de2757-03   /readwrite_partition  ext4   defaults    0 2
/readwrite_partition/var  /var  none bind 0 0
/readwrite_partition/etc  /etc  none bind 0 0
/readwrite_partition/home /home none bind 0 0
/readwrite_partition/srv  /srv  none bind 0 0

This raspberry pi is turned off by disconnecting the power, without doing a normal shutdown. I made the root partition readonly so there would be less chance of data corruption from shutdown.

If there is some problem and it is unable to mount the readwrite partition, it should use the versions of /etc, /home/, /srv, and /var from the readonly partition, and should be able to run disk repair utilities on the readwrite partition.

I have run this for two years and had no problems.

Other people make /var a seperate partition, and then /home a link to /var/local/home.

For /etc, only some of the files need to be readwrite. I made all of /etc readwrite because I was too lazy to figure out which files needed to be readwrite and which files could be readonly.

I included /srv in the readwrite partition because other people suggested it. But /srv is empty, so this does not matter.

I moved /etc/fstab to /fstab because I thought there might be problems if there were two versions of /etc/fstab, one on the readonly partition and one on the readwrite partition, and the two versions were different. The readwrite partition is mounted according to /etc/fstab on the readonly partition, but once the system is running, /etc/fstab on the readonly partition is hidden by /readwrite_partition/etc/fstab, and /etc/fstab on the readonly partition cannot be edited. I did not know if I would need to edit /etc/fstab, so I moved /etc/fstab to /fstab so that I would be able to edit /etc/fstab. If you are sure you will never need to edit /etc/fstab, then there is no need to move /etc/fstab to /fstab.

The hardest part is deciding how much configuration and edits to /etc you want to do before copying /etc from the readonly partition to the readwrite partition.

Most linux distributions do some configuration the first time it is booted, and also start making logs. You might want to do the first boot before copying /etc, /var, /home, and /srv to the readwrite partition, or you might want to copy the files before doing the first boot. The advantage of doing first boot before copying is that both /etc will include the first boot configuration. The advantage of copying before first boot is that the readonly versions of /etc, /home, /var, and /srv will preserve the original configuration and will not waste disk space on useless logs.