Measuring peak memory usage of many processes

Question

I have a bash script that calls various other scripts, one of which has a bunch of commands that launch scripts inside a screen session like this:

screen -S $SESSION_NAME -p $f -X stuff "$CMD\n"

Will running my top script with /usr/bin/time -v capture the peak memory usage everything? I want to have this script run as a cron job but I need to know how much memory it will take before I cause problems for other users on the machine.

Thanks

Answer 1

Yes and no, GNU time tries to show the summary/peak of everything.

You may check that using a small C program, say mal.c:

#include <stdlib.h>
#include <string.h>

int main(int argc, char **argv)
{
    long bytes;
    void *buf;

    bytes = atol(argv[1]) * 1024;
    buf = malloc(bytes);
    memset(buf, 0, bytes);
    printf("");
    return 0;
}

And see:

$ gcc mal.c -o mal
$ /usr/bin/time -f "%M" sh -c "./mal 5000"
5452
$ /usr/bin/time -f "%M" sh -c "./mal 10000"
10452
$ /usr/bin/time -f "%M" sh -c "./mal 5000; ./mal 10000"
10452

But as mentioned in the getrusage(2) man page (Linux), this is the max RSS of any single child process, not the instantaneous cumulative RSS of a tree of processes, all simultaneously using memory

In other words it does not summarize parallel or background processes, as you can see:

$ /usr/bin/time -f "%M" sh -c "./mal 1000000 & ./mal 5000"
5452

This also implies that you need to run time inside of your screen session to measure $CMD and not only screen.

FYI the difference between the shell built-in is that the time binary can't directly summarize pipes or functions:

$ time /bin/sleep 1 | /bin/sleep 2
 real 2.00
 user 0.00
 sys 0.00

$ /usr/bin/time -p  /bin/sleep 1 | /bin/sleep 2
 real 1.00
 user 0.00
 sys 0.00

Answer 2

(2025) If you are OK with a peak value that includes page cache usage, you can launch your processes in a container with a memory cgroup attached and look at the memory.peak cgroup memory interface file to get an approximate value:

# Max memory usage of two processes doesn't overlap
$ systemd-run --scope --unit=mempeak \
   bash -c 'python -c "import time; b = bytearray(100*1024**2)" & python -c "import time; time.sleep(2); b = bytearray(100*1024**2); time.sleep(1)" & wait; cat /sys/fs/cgroup/system.slice/mempeak.scope/memory.peak'
Running as unit: mempeak.scope; invocation ID: 43b16dd6be00418bb8606b8c66032cd4
113532928

# Max memory usage of two processes overlaps
$ systemd-run --scope --unit=mempeak \
   bash -c 'python -c "import time; b = bytearray(100*1024**2); time.sleep(2)" & python -c "import time; b = bytearray(100*1024**2); time.sleep(1)" & wait; cat /sys/fs/cgroup/system.slice/mempeak.scope/memory.peak'
Running as unit: mempeak.scope; invocation ID: a59c8efc605445b7a3e1d834be9ed11d
218447872

Other approaches (including variations on this one) are described in the How to measure peak memory usage of a process tree? question.