/dev/mapper/centos-root is at 100% Running CentOs

Question

we have a VM that host our docker and is currently out of space, I've tried everything on the net and nothing worked out yet, it is so full that I can't even run yum autremove

The system type is XFS, and I'm trying to fix this problem without having to remove the partition and create another one. My last solution if I couldn't figure it out is to recover the image from a previous vm version.

I apologize if I have any lack of knowledge in describing this issue and terminology, appreciate any help.

df -h shows below

Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                  24G     0   24G   0% /dev
tmpfs                     24G     0   24G   0% /dev/shm
tmpfs                     24G   18M   24G   1% /run
tmpfs                     24G     0   24G   0% /sys/fs/cgroup
/dev/mapper/centos-root   39G   39G   20K 100% /
/dev/sda1               1014M  197M  818M  20% /boot
/dev/sdb1                4.0T  275G  3.5T   8% /data/prom_tsdb
/dev/mapper/centos-home   19G   33M   19G   1% /home
tmpfs                    4.8G     0  4.8G   0% /run/user/0

ls -la from /dev/mapper

total 0
drwxr-xr-x  2 root root     120 Feb 27 17:10 .
drwxr-xr-x 20 root root    3280 Feb 27 17:10 ..
lrwxrwxrwx  1 root root       7 Feb 27 17:10 centos-home -> ../dm-2
lrwxrwxrwx  1 root root       7 Feb 27 17:10 centos-root -> ../dm-0
lrwxrwxrwx  1 root root       7 Feb 27 17:10 centos-swap -> ../dm-1
crw-------  1 root root 10, 236 Feb 27 17:10 control

lsblk -f outputs this

NAME            FSTYPE      LABEL           UUID                                   MOUNTPOINT
sda
├─sda1          xfs                         8f8242f9-7c24-4b84-a63e-bdf525dc1050   /boot
├─sda2          LVM2_member                 RoHLXd-Co1E-4UZ5-TeTD-JXx0-ySlG-hPjR7m
│ ├─centos-root xfs                         1329ba02-4819-43a6-b98c-64af4881b353   /
│ ├─centos-swap swap                        f9dfdb0b-1f24-45a9-939b-2ee49ace1461   [SWAP]
│ └─centos-home xfs                         33f236ac-07aa-4552-916c-1eb34d273fab   /home
└─sda3
sdb
└─sdb1          ext4                        c897a887-7bdc-4ba8-b134-f76a3e96efc4   /data/prom_tsdb
sr0             iso9660     CentOS 7 x86_64 2020-11-02-15-15-23-00

Answer 1

It looks like /dev/sda3 might be unused. If it's actually unused and big enough to be useful, you could add it to the centos LVM volume group (VG for short), and then use its capacity to extend /dev/mapper/centos-root. Also, if sda3 is not reusable for some reason, but there is unpartitioned space on the /dev/sda disk, now would be a good time to get it into use, by creating a new partition from the unpartitioned space.

To verify, use lsblk without the -f option, or fdisk -l /dev/sda.

You might have to clear a little bit of space first, to allow the LVM tools to successfully make their LVM configuration backups before each operation. In many cases, yum clean all does that nicely by cleaning the package management caches.

yum clean all
pvcreate /dev/sda3
vgextend centos /dev/sda3
lvextend -r -L <new size> /dev/mapper/centos-root

All this can be done while /dev/mapper/centos-root is mounted and in use. No reboot required.

If /dev/sda3 is not big enough and the sda disk is fully allocated (check with fdisk -l /dev/sda or lsblk without the -f option), then you might have to extend the sda virtual disk. How to do it depends on what you are using as a virtualization host. But once you've done that, using the new capacity requires some more steps.

First, install the cloud-utils-growpart package, as the growpart command in it makes it really easy to extend partitioned virtual disks.

Verify that the VM's operating system recognises the new increased size of the sda virtual disk, using lsblk or fdisk -l. If not, run echo 1 > /sys/block/sda/device/rescan and check again.

Now you can extend sda3 to cover the added space:

growpart /dev/sda 3

(or if sda3 is in use by something else, use fdisk to create a new partition sda4. Verify that lsblk sees it; if not, use partprobe /dev/sda to rescan.)

If you already added sda3 to the centos volume group, use pvresize /dev/sda3 to make LVM aware of the larger size of the partition, and then lvextend to add the capacity to the filesystem, same as above.