We have figured out a way to achieve this on our Linux DB server (as we can't afford to restart the DB server as it will affect so many users/DBs) and this has worked every single time so far :
We have NFS mounts over WAN !!
Check if problem is there and still exists:
sudo /usr/bin/egrep -i "nfs:|responding" /var/log/messages
Check processes we may need to kill
ps -ef | grep lsof | grep -v grep
ps -ef | grep 'df -k' | grep -v grep
ps -ef | grep 'df -h' | grep -v grep
ps -ef | grep "/bin/ls -ltr /" | grep -v grep
May have to kill these too? Be careful just see if anything manually needs to be killed as df or ls could be part of some other unrelated command/process
ps -ef | grep 'df' | grep -v grep
ps -ef | grep 'ls' | grep -v grep
ps -e -o s,pid,cmd | grep ^D | grep -v grep
ps -e -o s,pid,cmd | grep ^Z | grep -v grep
ps auxwww | grep -w Z | grep defunct | grep -v grep
Step 1. Unmount all NFS in lazy force fashion
Generate umount -lf commands for all the mount points.
grep : /etc/fstab | grep -v ^# | awk '{printf "umount -lf %s\n" , $2}'
Or
sudo mount | grep : | awk '{print "umount -lf " , $3 }'
You may have to run it several times the same "umount -lf " commands until you can run df -h successfully showing no NFS
This will help later as you can’t kill Defunc processes or kthread (pid 2) so by unmounting every NFS, we are forcing it to die in a few minutes once kthread comes in user mode from kernel mode and realizes that NFS is gone and kill itself (my understanding)….
Step 2. Kill all the possible processes (lsof, df commands, ls -ltr commands, Zombie processes, Defunc processes) that are still hanging while trying to use NFS mounts
ps -ef | grep lsof | grep -v grep | awk '{print $2}' | xargs kill -9
ps -ef | grep 'df -k' | grep -v grep | awk '{print $2}' | xargs kill -9
ps -ef | grep 'df -h' | grep -v grep | awk '{print $2}' | xargs kill -9
ps -ef | grep "/bin/ls -ltr /" | grep -v grep | awk '{print $2}' | xargs kill -9
Be Careful
Very Very Useful
As a last resort only
It tries to kill all Defunct and all Zombie processes so use it only as a last resort to avoid reboot of Db nodes
ps -e -o s,pid,cmd | grep ^Z | grep -v grep | awk '{print $2}' | xargs kill -9
ps auxwww | grep -w Z | grep defunct | grep -v grep | awk '{print $2}' | xargs kill -9
ps -e -o s,pid,cmd | grep ^D | grep -v grep | awk '{print $2}' | xargs kill -9
You can run the above commands several times until overall it looks better but the last one looking for Defunc processes is not possible to kill as we unmounted all the NFS mounts in step 1, eventually, they go away after 5-15 minutes and then you can attempt mounting NFS again. My guess is kthread process which is the parent process of the Defunc process with IP of NFS server, comes in user mode every few minutes, and at that time it realizes NFS mount is gone and fails/kills itself (but I am not a Unix expert :D )
##Last resort find parent processes of Zombie and check if you can kill them don’t kill pid 1 (pid 2 I tried killing didn’t work as you can’t kill it)
ps -e -o s,pid,cmd | grep ^D | grep -v grep
ps -e -o s,pid,cmd | grep ^Z | grep -v grep
ps auxwww | grep -w Z | grep defunct | grep -v grep
Find their parent and see if parent pid is safe to kill and would help or not kthread (pid 2) you can't kill!!:
find out parent pid
ps -ef | grep child pid
Wait after killing all lsof and df -k and df -h for few minutes and it started allowing the mount!!
After killing and waiting 5-10 minutes you can try mounting one of the mount point in verbose mode (-vvv)
mount -vvv one of the mount directory from above
If it hangs ctrl+C
And repeat Step2 again , wait 1 -2 minutes after killing zombie/defunc processes and try mount -vvv again.
Repeat until mount -vvv works for one of the mount directory
Once mount -vvv works for one of the mount directory you can mount all
Step 3. Attempt mounting all NFS mounts again:
mount -a
df -h
lsof -bhang?upstartand all. You probably want to restart all the services in thenfs-commonpackage, looks like there are a few. Order likely matters as well, so try stopping then starting in order of dependency. You probably also want to dorpcbindas your last stop/first start. I have done this before on Debian, but it just has one nicenfs-commonservice.