top/ps/mpstat showing different output

Question

When my top shows a load of 19/20 and my ps shows

root@server:/home/nohup# ps -eo pcpu,pid,user,args | sort -k 1 -r | head -5
%CPU   PID USER     COMMAND
122131 18695 mysql  /usr/sbin/mysqld  xxxxx
0.1   961 root     [kdmflush]
0.1   937 root     [kdmflush]
0.1  1025 root     [jbd2/dm-2-8]

But mpstat shows

root@server:/home/nohup# mpstat
Linux 2.6.32-5-amd64 (sms-db3)  06/13/2014  _x86_64_    (16 CPU)

09:35:42 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
09:35:42 PM  all    2.38    0.00    0.15    0.15    0.00    0.02    0.00    0.00   97.29

strace shows that mpstat is reading from /proc/stat to fetch the processor load.

Which one should I believe? What is causing this difference, or am I using it wrong? I am using Debian Squeeze 2.6.32-5-amd64

Answer 1

Debian Squeeze is running procps 3.2.8 which was a very old version of the library. mpstat shows numbers similar to top, so it is basically the numbers of the last few seconds. ps shows the stats from when the process started.

The numbers are unusual though which points to something overflowing or wrapping. If you want to work out what was happening right at this second, then mpstat seems more accurate. top should be close to those numbers too and if it isn't, I'd be curious to see what its summary area was showing.

PS. It is better to see how "busy" the process was across its lifetime.