Monitor processes trying to access non-existent file or directory

Question

We're moving websites from one server configuration to a new configuration and the websites will live in different paths than previously. We're planning to diligently go through and replace old paths with new paths, but in case we miss any, is there some way to monitor for any processes trying to access the old paths and also know what UID the process was owned by?

Answer 1

You can use this little systemtap script :

#!/usr/bin/stap
function proc:string() { return sprintf("PID(%d) UID(%d) PROC(%s)", pid(), uid(), execname()) }

probe syscall.open.return, syscall.stat.return,
  syscall.open64.return ?, syscall.stat64.return ? {
  filename = user_string($filename)
  if ($return < 0) {
    printf("failed %s on %s by %s\n", pn(), proc(), filename)
  }
}

It will hook the syscalls open and stat (you can copy/paste the code, maybe I forgot some other syscalls) at the return. As syscalls are the only way to communicate with the kernel, you won't miss anything. This script will produce this kind of output :

failed syscall.stat.return on PID(4203) UID(1000) PROC(bash) by /tmp/rofl
failed syscall.stat.return on PID(4203) UID(1000) PROC(bash) by /tmp/hihi

among the pros of using systemtap, we have :

• less instrusive for the process
• system-wide (not only the monitored process) but you can reduce its selection directly in the script
• less ressource hungry (only displays failed actions, not all to be grep after)
• you can improve the script to get the details about the calling program (eg, its backtrace, time of call, etc...). It depends on your application.

And for the cons :

• not standard, you have to install it (but standard enough to be available on most distribution). On Redhat & variants: sudo yum install systemtap
• need to have the debuginfos to build the module. On Redhat & variants : sudo debuginfo-install kernel

Some useful links : The tapset (included functions) index, and a beginners guide

Good luck for your migration !

Answer 2

Something like this should do it:

strace -f \
-e trace=open,stat,stat64,lstat,lstat64,chdir,mkdir,rename,symlink,creat \
 -o >(grep "the paths you want to catch" > log) \
commandToStartYourServer

You want the -f switch to track children processes. The trace options are a subset of what fabricate uses to trace IO (fabricate traces "open,stat,stat64,lstat,lstat64,execve,exit_group,chdir,mkdir,rename,clone,vfork,fork,symlink,creat")

This further limits disk IO by filtering the output via grep and process substitution (basically a pipe at the system level).

Answer 3

You could use fatrace. Kernel support (fanotify) should be present in most modern systems, but the fatrace userspace app is not in the standard repositories for some OSes. fatrace is particularly nice if your old files are on a separate file system.

http://www.lanedo.com/filesystem-monitoring-linux-kernel/

--

Far more common, though a little harder to use is auditd.

http://linux-audit.com/configuring-and-auditing-linux-systems-with-audit-daemon/

--

There is a fundamental problem with monitoring file usage by path when symlinks allow access via completely different paths, so you might need to pay special attention to that.