I am running a large simulation on a computer cluster using 50 compute nodes. This solver uses a data structure which grows on the file and differently for each node. I need to make sure the memory used does not grow beyond each node's memory limit.

So far, I am doing it in the most inefficient way: I have one terminal tab open for each node and run top to check the % memory used.

Is there a way I can do it with a script? The idea would be to ssh on each node and store the memory usage, ssh to the next, etc...