Virtualbox memory not reflected in its cgroup

Question

I've trying to limit the amount of memory a user can use using cgroups (using Ubuntu 16.04 with systemd).

After setting the MemoryLimit property of user-.slice I can see the change reflected in its cgroup's memory.limit_in_bytes.

However, when that user runs a VirtualBox VM, even though VirtualBox shows as using 2GB of RAM (see ps output below), that memory isn't reflected in the cgroup's memory.usage_in_bytes file:

$ cat /sys/fs/cgroup/memory/user.slice/user-1001.slice/memory.usage_in_bytes
1353863168

I can see the VirtualBox process id in the cgroups tasks file:

$ ps un -p 19678
    USER   PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    1001 19678 97.4 29.4 4660560 2378040 ?     Sl   07:25   3:48 /usr/lib/virtualbox/VirtualBox
$ grep 19678 /sys/fs/cgroup/memory/user.slice/user-1001.slice/tasks
19678

The memory.stat file for the cgroup shows the following values:

$ cat /sys/fs/cgroup/memory/user.slice/user-1001.slice/memory.stat
cache 423411712
rss 880754688
rss_huge 331350016
mapped_file 176910336
dirty 307200
writeback 0
pgpgin 14357052
pgpgout 14129610
pgfault 14533957
pgmajfault 16595
inactive_anon 546553856
active_anon 447401984
inactive_file 151785472
active_file 158326784
unevictable 36864
hierarchical_memory_limit 5368709120
total_cache 423411712
total_rss 880754688
total_rss_huge 331350016
total_mapped_file 176910336
total_dirty 307200

According to memory.txt, rss + cache should give a more accurate version of memory.usage_in_bytes. The result is 1304166400, which is even lower.

So why is VirtualBox's memory left out?

EDIT: added detailed output and memory.stat file.

Answer 1

I have a feeling that this document will help you a lot (Sections 5.5 and 5.2 especially). I will try to summarize as best I can here.

The best explaination that I have is that the numbers provided in memory.usage_in_bytes are not incredibly precise, but are intended to provide a general idea about the amount of memory something is using. memory.stat is likely to provide you with a more precise, albeit more complex, idea of the memory used by different processes. If the memory usage doesn't show up there, then you may have a much larger issue.

EDIT: If you keep reading in part 5.2 of that document, there is some additional information:

Only anonymous and swap cache memory is listed as part of 'rss' stat. This should not be confused with the true 'resident set size' or the amount of physical memory used by the cgroup. 'rss + file_mapped" will give you resident set size of cgroup. (Note: file and shmem may be shared among other cgroups. In that case, file_mapped is accounted only when the memory cgroup is owner of page cache.)

In this case that still doesn't work out to what you are seeing which leads me to believe that the owner of the virtualbox process is not the owner of all of the page caches in use by the virtualbox process.

The last sanity check I would advise would be to check the OOM killer log and see if the OOM killer has been active. When the OOM killer is active then you can get weird results from the memory logs.