Timeline for Kill a stuck process in running state
Current License: CC BY-SA 4.0
18 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 8, 2019 at 17:28 | comment | added | StackEng2010 |
@cas I have no NFS mounts, but I think the du hang may be related to docker containers hosted on this server.
|
|
| Aug 6, 2019 at 23:38 | comment | added | cas |
do/did you have any NFS mounts on that VM? If so, see nfs(5) and read the info on the soft and hard mount options (which control timeout & retry behaviour, effectively giving you a choice between hangs like this or the risk of data corruption). I ask because I've seen hangs on df or du many times over the years when an NFS server doesn't respond to an NFS client.
|
|
| Aug 6, 2019 at 22:23 | comment | added | StackEng2010 | Server restarted, a soft shutdown was stuck too. As this was an AWS instance, I had to stop the instance and start it back up again. Problem resolved, assumed kernel loop / bug. | |
| Aug 6, 2019 at 16:05 | comment | added | StackEng2010 | @mosvy I think you're right, that this is some kernel bug. Will restart the server later to resolve. Thank you | |
| Aug 6, 2019 at 13:05 | answer | added | shervin hariri | timeline score: -2 | |
| Aug 6, 2019 at 12:58 | comment | added | user313992 |
Which means that it cannot attach it (strace succeeds because it uses PTRACE_SEIZE which doesn't stop the process). This may be some kind of kernel bug -- also see this old question.
|
|
| Aug 6, 2019 at 11:58 | comment | added | StackEng2010 |
The last output of gdb -p PID is "Attaching to process 31806"
|
|
| Aug 6, 2019 at 11:54 | history | edited | StackEng2010 | CC BY-SA 4.0 |
Add process stats in text
|
| Aug 6, 2019 at 11:50 | comment | added | user313992 |
Could you attach to it with gdb -p PID? Anyways, please add the output of ps -Z PID (instead of the top screenshot, which may not be accessible to some people). Also mention if you're using any kind of virtualization/containerization.
|
|
| Aug 6, 2019 at 11:35 | comment | added | StackEng2010 |
@Henrik I am sending signals as root` @mosvy kill -TSTP PID doesn't stop it either.
|
|
| Aug 6, 2019 at 11:33 | comment | added | Henrik supports the community | A process can not catch a SIGKILL (signal 9) so if it doesn't help trying to send that, I can't see other possibilities than you not being allowed to send signals to the process at all. As the top output shows that it's owned by root, you need to be root to send it signals. | |
| Aug 6, 2019 at 11:33 | comment | added | user313992 |
will kill -TSTP PID stop it?
|
|
| Aug 6, 2019 at 11:26 | comment | added | StackEng2010 | @mosvy you're right, I've removed the mention of zombie state from my question. | |
| Aug 6, 2019 at 11:25 | history | edited | StackEng2010 | CC BY-SA 4.0 |
deleted 68 characters in body
|
| Aug 6, 2019 at 11:18 | comment | added | StackEng2010 |
@mosvy strace -e trace=kill kill PID returns exit code 0, and strace -p PID shows attached, but no output following kill, kill -9 command
|
|
| Aug 6, 2019 at 11:11 | comment | added | user313992 |
You can't kill a zombie, it's already dead. I have no idea what those many SE answers are talking about. You're probably rather thinking of processes stuck in an uninterruptible sleep (D, not R state). In your case: maybe you don't have permission to kill it? Try strace -e trace=kill kill PID, and if that shows that the kill() is successful, attach to the df process with strace -p PID, kill it and see how it's reacting to the signal.
|
|
| Aug 6, 2019 at 10:50 | review | First posts | |||
| Aug 6, 2019 at 11:20 | |||||
| Aug 6, 2019 at 10:48 | history | asked | StackEng2010 | CC BY-SA 4.0 |