10

I've faced a really strange issue today, and am totally helpless about it.

Some of the servers I manage are monitored with Nagios. Recently I saw a disk usage probe failing with this error:

DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied

I wanted to investigate and my first try was to check this directory permissions, and compare these with another server (who's working well). Here are the commands I ran on the working server and you'll see that as soon as I cd into the directory, its permissions are changed:

# Here we've got 555 for /sys/kernel/debug/tracing
root@vps690079:/home/admin# cd /sys/kernel/debug
root@vps690079:/sys/kernel/debug# ll
total 0
drwx------ 30 root root 0 Jul 19 13:13 ./
drwxr-xr-x 13 root root 0 Jul 19 13:13 ../
…
dr-xr-xr-x  3 root root 0 Jul 19 13:13 tracing/
drwxr-xr-x  6 root root 0 Jul 19 13:13 usb/
drwxr-xr-x  2 root root 0 Jul 19 13:13 virtio-ports/
-r--r--r--  1 root root 0 Jul 19 13:13 wakeup_sources
drwxr-xr-x  2 root root 0 Jul 19 13:13 x86/
drwxr-xr-x  2 root root 0 Jul 19 13:13 zswap/

# I cd into the folder, and it (./) becomes 700!!
root@vps690079:/sys/kernel/debug# cd tracing/
root@vps690079:/sys/kernel/debug/tracing# ll
total 0
drwx------  8 root root 0 Jul 19 13:13 ./
drwx------ 30 root root 0 Jul 19 13:13 ../
-r--r--r--  1 root root 0 Jul 19 13:13 available_events
-r--r--r--  1 root root 0 Jul 19 13:13 available_filter_functions
-r--r--r--  1 root root 0 Jul 19 13:13 available_tracers
…

# Next commands are just a dumb test to double-check what I'm seeing
root@vps690079:/sys/kernel/debug/tracing# cd ..
root@vps690079:/sys/kernel/debug# ll
total 0
drwx------ 30 root root 0 Jul 19 13:13 ./
drwxr-xr-x 13 root root 0 Sep 27 10:57 ../
…
drwx------  8 root root 0 Jul 19 13:13 tracing/
drwxr-xr-x  6 root root 0 Jul 19 13:13 usb/
drwxr-xr-x  2 root root 0 Jul 19 13:13 virtio-ports/
-r--r--r--  1 root root 0 Jul 19 13:13 wakeup_sources
drwxr-xr-x  2 root root 0 Jul 19 13:13 x86/
drwxr-xr-x  2 root root 0 Jul 19 13:13 zswap/

Have you got any idea what could causes this behavior?
Side note, using chmod to re-etablish permissions does not seems to fix the probe.

Improve this question
3
  • 2
    When providing samples of terminal input, consider replacing nonstandard aliases such as ll with the commands they stand for. Commented Sep 27, 2019 at 18:26
  • 3
    @RomanOdaisky It's a default alias in Ubuntu, might not have known it wasn't default Commented Sep 27, 2019 at 22:20
  • On top of what @tecloM said, this looks like a kernel bug for your kernel version. On 4.19.0-4 the behavior is normal. Commented Sep 28, 2019 at 22:52

1 Answer 1

Reset to default
20

/sys

/sys is sysfs, an entirely virtual view into kernel structures in memory that reflects the current system kernel and hardware configuration, and does not consume any real disk space. New files and directories cannot be written to it in the normal fashion.

Applying disk space monitoring to it does not produce useful information and is a waste of effort. It may have mount points for other RAM-based virtual filesystems inside, including...

/sys/kernel/debug

/sys/kernel/debug is the standard mount point for debugfs, which is an optional virtual filesystem for various kernel debugging and tracing features.

Because it's for debugging features, it is supposed to be unnecessary for production use (although you might choose to use some of the features for enhanced system statistics or similar).

Since using the features offered by debugfs will in most cases require being root anyway, and its primary purpose is to be an easy way for kernel developers to provide debug information, it may be a bit "rough around the edges".

When the kernel was loaded, the initialization routine for the kernel tracing subsystem registered /sys/kernel/debug/tracing as a debugfs access point for itself, deferring any further initialization until it's actually accessed for the first time (minimizing the resource usage of the tracing subsystem in case it turns out it's not needed). When you cd'd into the directory, this deferred initialization was triggered and the tracing subsystem readied itself for use. In effect, the original /sys/kernel/debug/tracing was initially a mirage with no substance, and it only became "real" when (and because) you accessed it with your cd command.

debugfs does not use any real disk space at all: all the information contained within it will vanish when the kernel is shut down.

/sys/fs/cgroup

/sys/fs/cgroup is a tmpfs-type RAM-based filesystem, used to group various running processes into control groups. It does not use real disk space at all. But if this filesystem is getting nearly full for some reason, it might be more serious than just running out of disk space: it might mean that

a) you're running out of free RAM,

b) some root-owned process is writing garbage to /sys/fs/cgroup, or

c) something is causing a truly absurd number of control groups to be created, possibly in the style of a classic "fork bomb" but with systemd-based services or similar.

Bottom line

A disk usage probe should have /sys excluded because nothing under /sys is stored on any disk whatsoever.

If you need to monitor /sys/fs/cgroup, you should provide a dedicated probe for it that will provide more meaningful alerts than a generic disk space probe.

Improve this answer
6
  • 1
    Thanks for this answer with as much details as I needed! I'll exclude /sys from my monitoring range. Commented Sep 27, 2019 at 12:21
  • 1
    @zessx: Also exclude /proc and probably /dev (because even if it isn't 100% RAM-backed, on the one hand it contains a number of files and directories that are "weird" in various ways, and on the other, if you actually consume a ton of disk space in /dev, your setup is horrifically broken and you should light the whole mess on fire). Commented Sep 28, 2019 at 2:23
  • "/sys is sysfs, an entirely RAM-based virtual filesystem" – I'm pretty sure the contents of sysfs are 100% synthesized from in-kernel data structures and do not live in RAM somewhere. In fact, I would argue that "RAM-based virtual filesystem" is an oxymoron: either it is RAM-based, i.e. has a backing store (even if it is a very non-traditional backing store for a filesystem), then it is not virtual, or it is virtual, then it has no backing store. Commented Sep 28, 2019 at 11:27
  • 1
    @JörgWMittag Note that I very carefully avoided saying sysfs is a RAMdisk. Where would the in-kernel data structures live, if not in RAM, anyway? I agree that the word "virtual" is problematic here, as you may be aware that on top of all the filesystem drivers in the Linux kernel is the VFS (Virtual File System) layer, which uses "virtual" in yet another sense, as an uniform abstraction for all possible filesystems. But it is hard to concisely describe just how proc and sysfs are different from real filesystems, as this was just background information to make the main point stick. Commented Sep 28, 2019 at 12:13
  • 1
    @grawity I tweaked the wording a little, would it be better now? Commented Sep 28, 2019 at 18:33

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.