Not counting the reserved space by root (usually 5%), if the /tmp is really 100% full, so 0 bytes free, why couldn't we log in to the machine via ssh? We only got the error message "cannot fork".
Are there any parameters to the ssh client so that we could still log in to a machine when it has a 100% full /tmp?
UPDATE: reboot is not really an option :\
I don't think that the full /tmp is the problem. The fork() system call only fails with that specific error if there are too many processes running.
Do you have a ulimit set?
If you can't access the console, then there's very little you can do...
I have seen cases where /tmp is backed by SWAP space....so if you are swapping AND have a lot of /tmp activity it is very possible that running out of SWAP space you would see this message.
If you are out of memory, then you may have very little choice than to reboot. The memory killer may be working, however, it indiscriminately kills off processes of which some might be important.
Another thing is that one (or more) processes have filled up the process table. With no more slots available, you can get that message as well. Your solution here would be a search and destroy operation on the processes causing the condition....assuming you can login. If you cannot login, then reboot is your only alternative.
Finally, of course, the machine software or hardware might be having a serious problem. Thus causing the situation you are describing.
On many systems, /tmp is a virtual file system that lives on RAM. Depending on how you're configured, filling it up may eat up all your RAM and leave you without enough memory to initialize a new process.
As you can imagine, that's not something that ssh can work around after the fact. But you can prevent it from happening in the future by tuning /tmp so that it can't grow quite this large.
fork: Cannot allocate memory
A process failing to fork is because either the system is out of memory or because the user has hit their maximum number of processes - because of, for example, a fork bomb.
I'm curious how you know that /tmp is full if you can't log into the machine - but my guess would be that whatever process has filled up /tmp may also be sucking up the resources needed for you to ssh in.
If you have physical access to the machine, you could try using a console login to get things back to a usable state, or use magic sysreq (assuming it's a Linux machine) to get things under control.
You may also be able to get sftp working even when ssh won't, since it ought to use less memory and fork fewer times than a shell - and you can use it to delete files from the remote, if you can get logged in.
As a workaround, try ssh after a reboot. Once I encountered similar problem which was resolved automatically after a system reboot!
/tmpandps -efwould be a great idea. With such an issue, your machine's services are very likely to go down anyway, which eventually makes the rebooting scenario quite enjoyable.