Is there a tool that allows logging of memory usage?

Question

I want to monitor memory usage of a process, and I want this data to be logged. Does such a tool exist?

Answer 1

Occasionally when the need arises I just do:

$ top -d 1 -b |grep <process> >>somefile

It's not an elegant solution, but gets the job done if you want the quick crude value to verify your hypothesis.

Answer 2

I have written a script to do exactly this. It basically samples ps at specific intervals, to build up a profile of a particular process. The process can be launched by the monitoring tool itself, or it can be an independent process (specified by pid or command pattern).

Answer 3

sar (System Activity Reporter) from the sysstat package is your friend in case like these.

Another way would be monitoring combined with historical data, e.g. Munin, pnp4nagios, rrdtools, ...

Answer 4

Besides the aforementioned sar, I'd recommend atop. It saves a binary log that you can peruse afterwards, and besides memory saves a lot of other information.

Answer 5

I want to monitor memory usage of a process, and I want this data to be logged. Does such a tool exist?

Yes, can achieve exactly this, a bit more with Procpath (author here).

In the simplest case you point it to the PID of interest and let it record its Procfs metrics including Resident Set Size (RSS). By default it makes a recording per 10 seconds (for the target process, its ancestors and descendants).

$ procpath record -d tgt.sqlite -p $PID

When you think it has recorded enough datapoints, Ctrl+C it and this will give you the recordings.

$ sqlite3 tgt.sqlite "SELECT ts, stat_rss * 4 rss_kib FROM record WHERE stat_pid = $PID"
1635349330.11141|478460
1635349340.12125|476068
1635349350.13124|480232
1635349360.14117|483416
...

Using a SQLite GUI can be a more convenient to explore and query the database (e.g. apt install sqlitebrowser).

Procpath can also visualise RSS of the target process into an SVG out of the box like this:

$ procpath plot -d tgt.sqlite -f tgt.svg -q rss -p $PID

If you're running Kernel 4.14+ you can get more advanced Linux memory usage metrics like Proportional Set Size (PSS), Unique Set Size (USS), proportional swap usage and others (see the documentation for what you can get from /proc/{pid}/smaps_rollup):

procpath record -d tgt.sqlite -p $PID -f stat,cmdline,smaps_rollup

Answer 6

You could try Valgrind.

Valgrind is an instrumentation framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail. You can also use Valgrind to build new tools.

The Valgrind distribution currently includes six production-quality tools: a memory error detector, two thread error detectors, a cache and branch-prediction profiler, a call-graph generating cache and branch-prediction profiler, and a heap profiler.

Answer 7

I like Sridhar's simple answer but I rolled my own before I tried his:

#! /usr/bin/python3

import json, psutil, datetime, sys, time

while True:
  print(json.dumps((datetime.datetime.now().isoformat(),
                    psutil.Process(int(sys.argv[1])).memory_info()._asdict())))
  time.sleep(1)

It's basically only useful if you want structured output. Note: _asdict() was broken in some versions of Python 3.5; it works again in Python 3.6.9 though (as well as in Python 2.7.17).

Output looks like:

["2019-03-19T11:21:53.784670", {"rss": 220389376, "vms": 538984448, "shared": 15724544, "text": 303104, "lib": 0, "data": 221364224, "dirty": 0}]
["2019-03-19T11:21:54.786136", {"rss": 220438528, "vms": 539119616, "shared": 15724544, "text": 303104, "lib": 0, "data": 221499392, "dirty": 0}]
["2019-03-19T11:21:55.787555", {"rss": 220495872, "vms": 539119616, "shared": 15724544, "text": 303104, "lib": 0, "data": 221499392, "dirty": 0}]
["2019-03-19T11:21:56.788754", {"rss": 220528640, "vms": 539119616, "shared": 15724544, "text": 303104, "lib": 0, "data": 221499392, "dirty": 0}]

For me, it was important to have structured output so that I could more easily consume it for analysis.