How do I profile a real world application?

Question

I run Debian 10 and since two weeks or so the PDF reader Atril (a fork of Evince) takes 25 seconds to start. Previously it started almost instantly. Now I'm trying to find out what causes the delay. I have downloaded the source package and built and installed it with profiling enabled:

cd "$HOME/.local/src"
apt source atril
cd atril-1.20.3
./autogen.sh
./configure CFLAGS=-pg LDFLAGS=-pg --prefix="$HOME/.local" --disable-caja
make V=1
make install

However, when I launch "$HOME/.local/bin/atril" no file named gmon.out is created. With verbose mode V=1 in the make command I can see that the option -pg is added to compilation and linking commands. Any clues? What's missing? There are several tutorials on the internet showing how to profile simple statically linked example programs but how do we profile "real world" applications?

Edit: It turned out that gmon.out was created in my home directory. However, when I run Atril through gprof the resulting output doesn't say much because the application is multi-threaded.

Answer 1

I personally wouldn't dig that deep. I'd recommend not building your own atril, but using the debian atril (because that's what you'll use afterwards, and also, debian gives you debugsymbols).

By order of complexity:

• run top (better even: htop, and in its settings disactivate "Hide userland process threads") while starting atril. Is the process using a lot of CPU? Go to perf or gdb.
• gdb is often good enough for such things:
  1. install the debug symbols for atril, see the debian wiki
  2. start gdb, loading atril: gdb atril (hint: gdb may tell you you're missing a few debug symbols – you can install these, too, so you can decipher backtraces more easily when they're not currently in atril)
  3. at the (gdb) shell, say run
  4. In the 15s of boredom, press ctrl+c. This interrupts the execution in gdb
  5. since atril is probably multi-threaded, info threads will show you were each thread is stuck
  6. Select the most interesting thread by its number N, by typing thread N
  7. get a backtrace: bt. You'll see where you're coming from.
• if it's not using CPU itself, it's almost certainly waiting for a system call to finish. strace atril will tell display the calls it does, live. what are the last few calls you get? maybe it tries to sleep?
• if you're CPU-bound, and generally, the perf command from the linux-base package is great: sudo sysctl -w kernel.perf_event_paranoid=-1, and then perf record -ag atril will sample regularly where the execution is stuck (but it looks at all processes, so close your browser, folding at home and whatnot), and then, perf report -g from the same directory shows you browsable statistics. These get more useful if you have debugging symbols installed.