removed lib modules folder after every reboot

Question

uname -r: 4.13.0-16-generic (ubuntu 17.10 server)

I have a problem that iptables doesn't work after a reboot:

iptables-apply /etc/network/iptables
[ ok ] Stopping fail2ban (via systemctl): fail2ban.service.
Applying new iptables rules from '/etc/network/iptables'... modprobe: FATAL: Module ip_tables not found in directory /lib/modules/4.13.0-16-generic
iptables-restore v1.6.1: iptables-restore: unable to initialize table 'filter'

Error occurred at line: 3
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
failed.
Error: unknown error applying new iptables rules: /etc/network/iptables
Reverting to old iptables rules... done.

I've tried to reinstall kernel and iptables:

apt install --reinstall linux-generic linux-image-4.13.0-16-generic linux-image-extra-4.13.0-16-generic linux-headers-4.13.0-16-generic iptables

when I reinstall it (without reboot) iptables works after a service restart.

when I reboot the machine I get the same error again.

I've notice that with every reboot the following folder is gone:

/lib/modules/4.13.0-16-generic/kernel/net/ipv4/netfilter

only 2 files left in folder ipv4 are: gre.ko udp_tunnel.ko

copying the missing files into this location doesn't work, will be removed with the next reboot.

Any idea how to fix this?

Answer 1

I get the same on Ubuntu 16.04. I don't have a fix, but I have some insight in to why this is happening. During the book process, the system initially boots on a RAM-based file system: initramfs. Later in the boot process, the hard disk takes over, but the modules directory from the initramfs is mounted on the modules directory of the hard disk. Note that the output of mount includes the line:

copymods on /lib/modules type tmpfs (rw,relatime)

Ok, so any changes to /lib/modules, e.g. by executing

sudo apt-get install --reinstall linux-image-$(uname -r)

will be lost when the system is rebooted. So how is this supposed to work? Well, when the apt install command adds new modules, it should put a corresponding entry in /usr/share/initramfs-tools/hooks. Then the installer script which apt install invokes runs update-initramfs, which generates the CPIO archive /boot/initrd.img-(uname -r) which is your initramfs image, which should now contain the modules needed for iptables to work when the system is rebooted.

The bug is that the relevant entries are never actually made in /usr/share/initramfs-tools/hooks, and therefor the relevant entries are not made in initramfs, and hence are not preserved when the system is rebooted.

At this point I am stuck.

Answer 2

As mentioned by @scottwh2, the issue is probably that a copymods ramfs volume is mounted at /lib/modules. To check this, run mount | grep /lib/modules.

To remove this, run

apt remove cloud-initramfs-copymods

Then reboot.

After rebooting, double check that the copymods entry is gone from mount.

Answer 3

I had the same problem on Ubuntu Server 16.04. The command:

mount | grep /lib/modules

displayed:

copymods on /lib/modules type tmpfs (rw,relatime)

So firstly I tried to remove cloud-initramfs-copymods, but that package was not installed in my system. Then I've install it:

apt-get install cloud-initramfs-copymods

And right after that I've remove it:

apt-get remove cloud-initramfs-copymods

And after rebooting my server copymods is gone and modules are not removed after reboot now! So thanks to scottwh2 and falsePockets I successfully resolved my problem.