2

After deleting files on a veracrypt-encrypted drive with no disk space left, no disk space is freed up.

I'm trying to sync a hard drive to another one as described here.

That target drive is full and I can't free disk space on it. I tried removing files in all sorts of ways such as running rm /.../file and it's not just the Dolphin file explorer that shows no disk space is free but also for example df -h which shows 0 in the column Avail and 100% in Use%.

How to actually delete the files and free up the disk space or make it show the actually available disk space if it's just displaying that wrong? I'm using Debian with KDE.

Details on how the problem occurred and more things I tried:

The problem was that it finished with errors because it reached "No space left on device" (which shouldn't have occurred either) and Grsync does not backup root-owned files.

I then ran the rsync command of Grsync in the console as also described above. The command is sudo rsync -r -t -p -o -g --delete -l -s /media/veracrypt1 /media/veracrypt2

It had many "skipping non-regular file" errors but that is a separate problem (it should backup 100.0%) and eventually also got a No space left on device error despite that there are many GBs of disk space free on disk1. sudo lsof | grep {diskname} shows disk2 is not used by anything. Of course there is no running rsync process. I rebooted and dismounted and remounted the drive in the meantime several times.

The problem I have that when I delete something on disk2 to free disk space to allow for another sync, it doesn't show that more disk space is available in the Dolphin file browser or when entering lsblk -f or df (the latter still shows 0 in the column Available).

I tried deleting it by right clicking the file in Dolphin and choosing delete. I also tried "Move to Trash" but there is only the backed up Trash folder (I can't exclude it with Grsync) but no Trash folder at the root directory of the drive.

A similar problem occurred on my /home/ partition (different drive) but it was solved after a reboot. The problem described here did not go away after a reboot, dismounting and remounting the drive, and after upgrading to Debian12.

When deleting things, the free space is still/immediately 0 bytes, nevertheless I checked for a large quickly growing file with gdmap but couldn't find anything. After the upgrade it seems like sudo lsof | grep "/media/" (no output even if I have some file open) doesn't show if files on the disk are opened anymore but no file on it should be open.

I already tried things from this question. For example IUse% shows the percentage is quite low. sudo dumpe2fs /media/veracrypt2/ | grep -i reserved just says dumpe2fs: Is a directory while trying to open /media/veracrypt2/ and sudo lsof +L1 | grep media shows no output and neither does sudo lsof | grep deleted | grep media. Nothing should write to or read from that drive.

Unnecessary details to skip:

I thought logging out and back in to restart the session would solve this but it didn't and it still shows 0 bytes of disk space free. rm filename does not free up disk space either. As requested the output of lsof | grep deleted is below; many of those messages showed many times and I left out some columns and replaced Ids:

pulseaudi ... /memfd:pulseaudio (deleted)
ksmserver ... /home/username/.cache/ksycoca5_{lang}_{id} (deleted)
plasmashe ... /home/username/.cache/ksycoca5_{lang}_{id} (deleted)
plasmashe ... /home/username/.cache/appstream/appcache-{id}.mdb (deleted)
vorta ... /usr/share/mime/mime.cache (deleted)
{texteditor} ... /home/username/.cache/ksycoca5_{lang}_{id} (deleted)
konsole ... /home/username/.cache/konsole/#{number} (deleted)

after restarting the session there is an additional .lock file open by a printer program, the two pulseaudi, and the konsole ones but nothing else. I still can't free up disk space if lsblk -f and Dolphin display the correct available space.

I tried lsof +L1 another time and it doesn't show more disk space even when this is all it shows (with one open konsole):

COMMAND       PID     USER   FD   TYPE DEVICE SIZE/OFF NLINK    NODE NAME
konsole    549098 username   20u   REG  254,2        0     0 8257752 /home/username/.cache/konsole/#8257752 (deleted)
konsole    549098 username   21u   REG  254,2        0     0 8257764 /home/username/.cache/konsole/#8257764 (deleted)
konsole    549098 username   22u   REG  254,2        0     0 8259787 /home/username/.cache/konsole/#8259787 (deleted)
konsole    549098 username   23u   REG    0,1  3677184     0 2372415 /memfd:wayland-cursor (deleted)
veracrypt 2340184 username    9u   REG    0,1  3677184     0   69165 /memfd:wayland-cursor (deleted)
xdg-deskt 2340702 username    8u   REG    0,1  3677184     0   69892 /memfd:wayland-cursor (deleted)

I also tried sudo lsof | grep deleted | grep mountname and it does not give any output. mountname is part of the dir since I mounted it there (sudo lsof | grep mountname1 works for files open that are on the other drive and even sudo lsof | grep mountname does not show anything).

This could be a veracrypt bug, issue is here.

Improve this question
15
  • 2
    By diskname, do you mean the directory to which your are trying to rsync the data (disk2) or the name of an actual disk? Commented Jan 19, 2024 at 0:30
  • 1
    I deleted it by right clicking the file in Dolphin and choosing delete. GUI file applications often don't remove files, instead moving them to a 'trashcan' where they can be recovered. What happens when you pick a file that's large enough to be noticed and use rm filename to delete the file? Commented Jan 19, 2024 at 4:46
  • 2
    You're grepping wrong. lsof shows files (file descriptors to be more correct) and sockets, not partitions. Commented Jan 19, 2024 at 12:19
  • 1
    Add the output of lsof | grep deleted to the question. Commented Jan 19, 2024 at 12:40
  • 1
    @NasirRiley I have added that. Commented Jan 24, 2024 at 14:36

3 Answers 3

Reset to default
2

I had this scenario run on me several times on different machines. I cannot guarantee you've stumbles on the same conditions I did, but I believe it's worth checking out.

You seem to be having two distinct problems, none really connected with Veracrypt which is only the container for a file system.

  1. The disk filled up, and deleting some files resulted in the disk still being filled up.

  2. The disk should not have filled up.

1. disk full

Case (1) does not depend on deleted files or unaccounted space because you remounted the disk and rebooted, which would have got rid of deleted-open files and forced a fsck should it have been necessary.

1.1 disk full because something keeps writing to it

We are then left with two solutions: one is that something is still writing to the disk, appending to some existing file (it usually happens to me with log files). Imagine a process has a backlog of twenty megabytes that need to be written, it cannot because the disk is full, and you delete fifteen megabytes of files. Your free space is immediately filled up again. To check on that, the only practical way is to run a disk accounting (to a file on a different device), ls -laR might do, delete some files to free space, run the accounting again, and compare the two accountings. You should of course notice that some files on the second have disappeared (the ones you deleted), but also that some other files have appeared or grown in size.

You can do this using lsof to only account open files, which is much faster, but you risk missing files that are created, filled and closed.

You can also do this using du -sk /mnt/disk/*/*/* > /tmp/before.txt to list space in all folders of, say, depth up to 3 from the mount point of the disk. This allows to quickly zero on which folders are increasing in size.

1.2 disk apparently full because of 5% root reservation

The second possibility is that your file manager does not properly account for root reserved space. Most if not all Linux file systems reserve some space (typically 5%) for root operations and to increase some key performances. If a root-instigated operation fills the disk, and a non-root user deletes some files remaining above 95% of occupancy, the non-root user will keep seeing the disk as completely full.

You can check total filesystem and available space using

tune2fs -l /dev/mapper/yourcryptdevicename

and you can change, say, 5 to 1 percent with

tune2fs -m 1 /dev/mapper/yourcryptdevicename

which will make the remaining 4% free space "reappear" to non-root users.

2. Disk should not have filled up

This only applies if you placed, say, 400 GB of files on a 500 GB disk and filled it up. If the difference is below 5%, then it is an effect of cause 1.1 and can be solved that way, no further explanation is necessary.

Otherwise, space on disks is allocated in multiples of the basic block unit, which might be 4K. When this happens, a file 6K in size will require two blocks, and creating it will result in the disk losing 8K of available space. This is sometimes called slack space and is on average equal to half a block size multiplied the number of files you have.

So when I sync my source and library tree from my laptop to a NAS, where the laptop has 2K blocks and the NAS has 4K blocks, there are 49 Gb actually used in 1,797,479 files; but if I run du -sk --apparent-size, I get 44Gb . The 49 Gb is actually thanks to some tricks the ext4 filesystem plays to avoid small files to gobble lots of slack space. On the NAS I do not have those tricks and blocks are double to start with, and those same 44 Gb of files take up 56 Gb. Scaling up to a whole SSD worth, I might have 880 Gb of files that do not fit on a 1 TB SSD drive.

To check whether this might be the case, run

tune2fs -l /dev/yourpartition | grep "Block size:"

There are other information in tune2fs output that might help you check what is happening, by comparing the output on the two disks you're syncing, and also comparing the output on the same disk after deleting some files, for example

Inode count:              365985792
Block count:              1463919616
Free blocks:              72775504
Free inodes:              176130892
Improve this answer
10
  • The ls command is a good idea, it would need to be run with sudo to have all access. In my case however, the drive is new where nothing is configured to write anything to it (only thing I did with it is the rsync commands) and I mounted it to different points with VC (/media/veracrypt3, 4, and so on). I checked for large files with gdmap and ran lsof | grep veracrypt2 during deleting large files. I also deleted files/dirs so large it would be hard to fill up so quickly. Haven't tried the du cmds yet as they run too long but will try more later (doubt).Both are ext4 filesystems & same model Commented Apr 14, 2024 at 11:37
  • Okay, excluding an error in the mount sequence, the blocks can only be allocated to a clean, committed file (visible to ls) or to a temporary, deleted file (visible to lsof). Maybe force a fsck on the unmounted filesystem. If the system sees it as used, the space must be allocated to something and it ought to be possible to have it tell us exactly what. You could mount the FS readonly and run a ls to check on the files at rest, as it were. Commented Apr 14, 2024 at 13:26
  • 1
    One other possibility is that you might have 5% space unavailable since it was reserved for the root user. As with fsck, once you decrypt (open) the volume you ought to see it as /dev/mapper/something. From a root command line, umount /dev/mapper/whatever will unmount the disk keeping it unencrypted. Now fsck /dev/mapper/whatever will work, but we don't care: Check Filesystem did exactly that and already found nothing. The important thing is that now tune2fs -l /dev/mapper/whatever will also work, and check the reserved block count, and even change it using tunefs -m or -r. Commented Apr 14, 2024 at 23:02
  • 1
    If FSUSE is between 95% and 100% and FSUSE is 0, I'm going to keep suspecting root's reservation of 5% disk space. With FSUSE below 94% and FSUSE 0, I'm at my wits' end about what could be the cause; the rest all has, but this has never happened to me. Commented Apr 18, 2024 at 6:42
  • 1
    Finally it shows some disk space! You were right about the 5% reserved disk space. I tried deleting more things until it's below 95% FSUSE. The other drive which I tried to copy over is below 95% FSUSE and seems to have more than 1% of disk space available. Do FSAVAIL and free disk space in Dolphin not consider root reserved disk space? I have now run sudo tune2fs -m 1 /dev/mapper/veracrypt2 to change 5% to 1%. Also the sudo du -sk cmd only showed 4 as filesize for some full dirs. I'll mark this as solution but please edit it to add this solution to the top. Commented Apr 20, 2024 at 11:35
2
+50

Information taken from df in linux not showing correct free space after file removal, and combined with other answers, Google results, and experience.

Your mountain of commenters is only providing you half the answer. lsof |grep deleted only lists the affected files, but quoting Ignacio of the link above:

Deleting the filename doesn't actually delete the file. Some other process is holding the file open, causing it to not be deleted; restart or kill that process to release the file.

Use:

lsof +L1 to list all processes using a deleted file (which you've done per the comments, but the comments aren't telling you why you did it). lsof is short for LiSt of Open File descriptors. The key word here is descriptors. We need the file descriptors in order to kill the process that is holding the now deleted file open. You can either use kill - 15 PID which will kill the process (the PID should be listed in the output in the ellipses above)[See Also: When should I not kill -9 a process?], or you can kill the open file descriptors like so:

  1. cd /proc/PID/fd: once again the PID will be listed as a number in the ellipsis. Replace PID in the command with that number. fd is short for File Descriptor
  2. Verify you are in the correct PID's branch of the proc filesystem with a variation command you were given by the commenters: ls -l |grep deleted
  3. Now issue >/dev/null 2>&1 This effectively writes the (deleted) file by redirecting Standard Error 2> and& Standard Output 1 to the null device. See also: How to redirect the output of a pipe to >/dev/null for a great explanation on dev/null and redirection.

Some readers may be asking why there is nothing written on the left-hand side of the first redirection operator and I believe because we are in a File Descriptor directory the file descriptor is assumed to be the lefthand argument.

Improve this answer
9
  • This be helpful for some people who have the same problem but it doesn't solve my problem: I already clarified in the question that no file on that partition/drive is shown by lsof +L1. Commented Apr 9, 2024 at 21:45
  • It's not going to show a partition or drive. It only shows Opened Files You need to compile a list from the output of that command and manipulate each PID separately You can't do it "by disk" It doesn't matter that the file that's keeping the process running isn't on the disk that you're trying to copy, backup, or move. You must stop the offending processes before you can complete the rsync Commented Apr 9, 2024 at 21:49
  • Continuing: Your other option is to use your Service Manager, most likely SystemD's systemctl to stop your shutdown your Desktop Environment and attempt the rsync command in a terminal, and after succeeding, restart the Desktop Environment. Commented Apr 9, 2024 at 22:00
  • I'm not running any rsync command now. Why would any file be held open when I dismounted, remounted the drive and rebooted in between? No file on the drive is shown with the commands that show open deleted files and in regards to the processes of lsof +L1 just randomly quitting processes like Xwayland doesn't seem like a good idea. I'm not looking to run any rsync command, I'd just like the drive to update its free disk space. Commented Apr 9, 2024 at 22:02
  • Because in order to "run" your Desktop Environment, the processes involved (like X/Wayland) all have to write to temporary files or sockets every tome something changes. Those temporary files are constantly overwritten each time the Desktop Environment is updated etc... It doesn't matter that you unmounted and remounted/restarted because the temporary files are either recreated or appended to after the reboot when the Desktop Environment restarts.. If your overall goal is to free up space on the encrypted drive, you can't just copy everything, as you've found. Commented Apr 9, 2024 at 22:16
1

I don't believe the veracrypt layer has anything whatsoever to do with the disk space issue. Veracrypt is a block layer that knows nothing about filesystems. Ext4 is a filesystem that sits on top of a block device. If you're not getting disk space back then it's a filesystem issue.

Considerations other than block size (discussed an in another answer) are sparse files and hard linked files being copied as separates. Both are managed by flags to rsync; both are ignored by your selection of flags. You can validate this by comparing the disk usage from these two commands:

du -hsx /media/veracrypt1
du -hsxl --apparent-size /media/veracrypt1

In your linked question you talk about backing up an entire system. In such cases you need to be careful to ignore memory-backed filesystems such as /dev and often /tmp. However, here you're using directory paths so I will continue to use those here

rsync -axHS /media/veracrypt1/ /media/veracrypt2

Notes:

  1. The sparse flag will not replace existing files with sparse equivalents - you would need to process those yourself or delete them first.
  2. Likewise rsync will not hard link files that are the same but not already hard linked; you would have to delete both beforehand.
  3. Your original command would have created /media/veracrypt2/veracrypt1. By suffixing / to the source directory path you perform a copy directly to the destination without first creating a subdirectory named as the source.
Improve this answer
1
  • The du commands take very long so I can't easily run them...what would be the conclusion if the output of the two commands differ? Or should I run them before and after deleting files? Rsync is not running anymore and also there aren't any /dev and /tmp files on it except maybe if they had been backed up to that drive from my root partition in the past. This answer seems more about the optimal rsync drive-sync command at the other question rather than the missing disk space so it would be better placed there. I'm not surprised by problems like that without a proper GUI for syncing. Commented Apr 14, 2024 at 15:16

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.