How to make the system automatically kill some processes?

Question

Sometimes my computer starts to behave sluggishly after running too many programs/processes simultaneously, at points almost looking crashed/frozen. Using Debian Linux, is there a way to automatically kill some processes before memory gets too scarce for smooth operation?

Answer 1

Basically, you want a daemon that monitors the free memory, and if it falls below a given threshold, it chooses some process and kills them to free up some memory.

while (true) {
    size_t free_memory = get_free_memory();
    if (free_memory < free_memory_threshold) {
        pid_t pid = choose_a_process_to_kill();
        kill(pid, SIGTERM);
    }
}

An obvious question is: how do you choose processes to kill? An easy answer would be the one with the biggest memory usage, since it's likely that that is the misbehaving "memory hog", and killing that one process will free up enough memory for many other processes.

However, a more fundamental question is: is it really okay to kill such a process to free up memory for others? How do you know that the one big process is less important than others? There's no general answer. Moreover, If you later try to run that big process again, will you allow it to kick out many other processes? If you do, won't there be an endless loop of revenge?

Actually, the virtual memory mechanism is already doing similar things for you. Instead of killing processes, it swaps out some portion of their memory to disk so that others can use it. When the former process tries to use the portion of the memory later, the virtual memory mechanism swaps in the pages back. When this is happening from different process contentiously (which is called thrashing), you need to terminate some processes to free up the memory, or more preferably, supply more memory. When the system starts