5

My linux server reports high disk space usage for device /dev/sda4 mounted on / as shown below: 

[root@stormtrooper03 /]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda4       126G  114G  5.5G  96% /
tmpfs            32G     0   32G   0% /dev/shm
/dev/sda2       239M  118M  109M  53% /boot
/dev/sda1       150M  264K  150M   1% /boot/efi
/dev/sda5        63G   52M   60G   1% /home
/dev/sda6        63G   54M   60G   1% /tmp
/dev/sda7        63G  2.6G   58G   5% /usr
/dev/sda3       539G   11G  501G   3% /var
/dev/sdb1       917G  857G   51G  95% /data/1
/dev/sdc1       917G  861G   47G  95% /data/2
/dev/sdd1       917G  282G  627G  31% /data/3
/dev/sde1       917G  861G   47G  95% /data/4
/dev/sdf1       917G  858G   50G  95% /data/5

However, I know there is much less than this amount used by running du -sh /* as shown below. I need to clean files setting on this device but I can't find them!!

[root@stormtrooper03 /]# du -sh /*
6.7M    /bin
116M    /boot
3.7T    /data
356K    /dev
30M /etc
172K    /home
638M    /lib
20M /lib64
16K /lost+found
4.0K    /media
4.0K    /mnt
3.8G    /opt
0   /proc
42M /root
9.2M    /sbin
4.0K    /selinux
4.0K    /srv
0   /sys
2.1M    /tmp
2.5G    /usr
11G /var

Update I have found reason for this: a new disk was mounted on /data/3 which had 110GB of data and became hidden from the filesystem after mounting a disk on it.

The answer to the question why does mount happen over an existing directory explains how to uncover original data using mount --bind but it doesn't explain how to find if your system has such "shadowed" directories if you don't know ahead of time.

Is there a quick way to a list of disks mounted on non-empty directories without checking each one individually by mount --bind?

Improve this question
1
  • /var is on a different device (/dev/sda3) and even if it was on the same device it's only 11G while df reports 114G usage Commented Feb 18, 2017 at 2:21

3 Answers 3

Reset to default
1

I figured out the problem! By tracing actions on this server that are different from other servers I found that at one point the disk mounted on /data/3 stopped working so it was commented out in /etc/fstab. However processes in the system didn't know about this and continued to write to /data/3. which at this time is a directory in /. Then later the bad disk was replaced and the fstab entry was updated so now /data/3 is mounted on the new disk. Somehow the old data was hidden as it seems there is no pointer to it or something.

Once I knew this, I unmounted the disk /data/3 and now I was able to see 110GB of data setting there which I deleted and then remounted the disk and things are back to normal.

I don't know if there is any tool to find such orphan directory so to speak.

Improve this answer
2
  • I have heard of this as "shadowing" a directory. It is often done on purpose when mounting volumes within containers. I am not aware of a program that would have detected this, unless there is a special flag for it in du. Maybe re-word the question in order to attract people that may know a way of discovering the issue sooner, or who know of a method of detecting shadowed directories more readily. Commented Feb 20, 2017 at 19:47
  • Thanks for the term "shadowing". I have updated the question for future readers. Commented Feb 20, 2017 at 20:36
0

[EDIT]: Leaving this answer as a general note, it does not answer the refined version of the question.

KDirStat is a good graphical utility for seeing where large files and directories are on a system. It gives a heatmap and a sorted list.

Improve this answer
4
  • This could not uncover those hiding files either. Commented Feb 18, 2017 at 3:06
  • @MajidAlfifi, did you run it as root and tell it start from /? If I remember correct, by default it only analyzes the user's home directory. Commented Feb 20, 2017 at 19:37
  • 1
    KDirStat will show the mount point as such, and only using the size of 1 inode when the shadowing device is mounted. It will show the real used space if the device is unmounted. Same results, only graphical, as du. Commented Feb 20, 2017 at 21:42
  • How about: du -ak | sort -n Commented Feb 22, 2017 at 19:20
0

If you mount the file system that might have the hidden data elsewhere, then you can go within the location to the relative spot where mounts might hide the data and see if there any any data there.

If sudo ls -A $MOUNTPOINT produces any results, then there is something there at $MOUNTPOINT.

To see what's mounted run the mount command by itself.

Here's some code that you can use to look for hidden data.

#!/bin/bash

TMP="/media/root" # temp location to mount the directory that might have data hidden by a mount point
sudo mkdir -p "$TMP"
sudo mount --bind --read-only -- "/" "$TMP" # if "/" holds the mount point with the hidden by mountpoint data, replace "/" if needed.

# replace `$(mount | awk '{print $3}')` below with the directories you wish to test for hidden data
for LOCATION in $(mount | awk '{print $3}') ; do
    if [ -n "$(sudo ls -A "$TMP$LOCATION" 2>/dev/null)" ]; then 
        echo "data on unmounted \"$LOCATION\"";
    fi; 
done;
sudo umount "$TMP"

sample output:

data on unmounted "/run"
data on unmounted "/"
data on unmounted "/var"
Improve this answer
3
  • I was hoping there maybe a way to check the whole system without unmounting/mounting every disk one by one. Commented Feb 21, 2017 at 23:24
  • Just saw this after my post, maybe it'll help using --bind. unix.stackexchange.com/questions/267629/… Commented Feb 21, 2017 at 23:27
  • @MajidAlfifi I've changed the code so the directories don't have to be unmounted. Commented Feb 22, 2017 at 4:52

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.