I have a CentOS7.9 system showing 56GiB free RAM (free -m) but only 1MiB free swap, and it's been in this state for three days. The original problem report was that a large (EE simulation) app keeps crashing.
Can anyone help me understand what could put the memory into this condition?
You don’t mention how much swap is actually used, but unless there’s a huge amount, this sort of situation can be explained by a process (or multiple processes) consuming all the available memory, or causing it to be consumed (e.g. by reading many files without indicating that they shouldn’t be cached). This causes as much data as possible to be pushed out of memory, and some of that will end up in swap.
Data in swap is only pulled back into memory if something needs to use it, so it’s common for systems to accumulate data in swap (corresponding to data which was used at some point but isn’t regularly required). Even when data is used, it can also be kept in swap — so that if memory needs to be freed again, it doesn’t need to be copied back to swap (if it hasn’t changed).
The only way to reliably reduce swap usage is for processes with data in swap to exit (or be killed), or for the available swap itself to be reduced (swapoff).
The sort of situation you describe shouldn’t cause insurmountable problems. If the data in swap is actively being used, the system will slow down quite a bit until it’s reloaded into memory, but since you have lots of available memory your system is nowhere near thrashing (where it spends all its time shuffling data to swap and back).
As this large app fills memory other parts of "inactive" software will be pushed out to the swap area (/proc/sys/vm/swappiness can be changed to affect how aggressively it moves things from RAM to swap). When it it finally exits/crashes then all the memory is freed, leaving a bunch of other software in the swap area and a bunch of free RAM.
The memory won't be moved back out of the swap until whatever application it belongs to uses it. If the memory sitting in swap isn't used very often it might take a while to be pulled back to RAM. You can turn the swap off (swapoff -a) to force it to flush everything back to RAM and empty the swap (then swapon -a to turn the swap back on).
swappiness doesn’t determine how aggressively things are moved from RAM to swap, it determines whether pages are evicted from the working set or the page cache.
sudo swapoff && sudo swaponwhich will turn off and then reanable swap and that should clear it out. But in principle (at least as far as I know, but take this with a grain of salt, I am not 100% sure) as soon as you need swap, the old stuff should be swapped out.