0

Apologies in advance if there is an existing answer I should consult...

Fired up Kodi for the first time on my Ubuntu Studio 24.04 box. It has a 1 TB hard drive, of which almost half is consumed by the .flac files. Before running Kodi, about a quarter of the disc was free.

I went through and added the appropriate media directories to to kodi's configuration, and initially it seemed to be working.

Came back this morning, selected music, and rather than coming up with the list of albums Kodi locked up, unable to write log files to a full disk.

Killing Kodi confirms disk is full, but it isn't clear what or where the additional space was consumed. /home, including the media, still reports under 500GB. Going through other directories under / with sudo du, I'm not finding anything over 12 GB (for /usr). So I'm very confused about where the space went.

Admittedly ls | xargs sudo du -sh executed from / reports that it can't access things under /proc/117484 or /run/user/1000/doc.

Has anyone else seen this, or does anyone have a suggestion for where to look for the missing gigabytes?

(No, I haven't tried just rebooting yet; I'm a trifle afraid to do so before I know I have enough disk space to run safely. )

1
  • I'll take a look. It's also been suggested that I check lsof since open files may not be reported in directory totals. Commented Feb 26 at 19:18

1 Answer 1

0

Found the problem and it wasn't Kodi's fault; self-inflicted wounds, specifically mount mistake and backup misconfiguration. Documented below is how I pursued it and what I found, just so folks can see my debugging process:

Since I had run out of disk space in which to hold and sort output of du / and lsof it was a bit hard to analyze the results; I could grep them but didn't have space in disk to sort them. However, the Emacs editor can do some in-memory sorting, and a combination of that and some programmed editing convinced me that the problem wasn't something like huge files still being held open.

I rebooted into a "live" session on my install media and investigated from there.

Mounted the /dev/sda2 partition (my usual root filesystem) as /mnt/tmp and ran file system check against it. (Reminder, never run fsck against a file system which is actually in use, and certainly not against the one you booted from!) fsck reported no errors, so whatever was consuming space seemed to be legitimate files. The question was where and why.

Since I had space in the live environment for the scratch file, I could run ``du /mnt/tmp | sort -n` to get cumulative disk consumption by directory. Looked at the highest users, and there it was: The directory where I normally mounted my backup drive (but hadn't done so in the live session) was reporting hundreds of gigabytes if data when, without the filesystem mounted, it should have been empty.

My error was that my backup program (dsnapshot) have been configured to copy file system changes to a subdirectory at the mount point. But unlike windows, Linux does not actually mount USB drives by default. There are some conditions under which the gui "magically" mounts whatever was previously mounted, but that wasn't happening for the backup tool running as a background cron job. As a result, the backup was happening to the actual directory at the mount point rather than to another driver's filesystem mounted there... And so was copying the root file system's contents into a directory inside the root file system, with the obvious consequences of almost immediately filling the disc with those misplaced copies.

I haven't picked this up when I had tested backup because I hadn't rebooted since I last manually mounted the backup disc, and the copying went where it was supposed to go. But the first time I rebooted, that mount was no longer present with the effects mentioned above.

Some of the documentation forrsnapshot actually mentions this hazard in backing up to a USB file system; the copy I found happens to be at https://wiki.archlinux.irg/title/Rsnapshot but I'm sure this can be found in the Ubuntu docs or in those for the rsnapshot git project. Basically they involve first editing the /etc/fstab file so the drive can be automatically mounted, then altering the backup tool's `/etc/systems/system/[email protected] file so that mount occurs before the backup process begins copying files.

They also recommended specifying the backup's target directory using the file system UUID, to make sure that what is mounted at that directory really is the file system you want to copy into. That would have helped prevent the misplaced backups when the mount hadn't occurred.

I am applying those changes now, of course. I haven't yet fully tested them, but they make sense as an explanation of what I did wrong and how to do it right. I'll try to remember to update this after complete confirmation of the fix.

("Did you plug it in and turn it on?" Oops. Obvious in retrospect. Experience is that which enables us to recognize a mistake when we have just made it again.)

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.