How to list all the processes assigned to a user on a cluster

Question

We have a cluster in our university controlled by SLURM. The issue is that I have observed sometimes some of the resources are not being realised even though they do not show up in the squeue command. For example I have seen that a lot of CPUs on one of the nodes are assigned to me despite the fact that I had killed the allocated resources with scancel a couple of days ago! I want to find them and kill them.

On my local machine I have generated public and private ssh keys with ssh-keygen so now I can login to any of those machines with ssh foo, ssh [email protected], ssh [email protected] ... but the names of those nodes are not in an order. If one login to one of those nodes and run sinfo this is the result:

PARTITION   AVAIL  TIMELIMIT  NODES  STATE NODELIST
debug          up   infinite      3    mix baz[080-081,083]
debug          up   infinite      2  alloc grault,baz082
debug          up   infinite     13   idle baz[061-070],corge,bar,quux
gpu_p100       up   infinite      1    mix baz080
gpu_titan-x    up   infinite      2    mix baz[081,083]
gpu_titan-x    up   infinite      1  alloc baz082
r730           up   infinite      1    mix baz080
t630           up   infinite      2    mix baz[081,083]
t630           up   infinite      1  alloc baz082
r930           up   infinite      1  alloc grault
m610           up   infinite     10   idle baz[061-070]
r720           up   infinite      1   idle corge
r815           up   infinite      1   idle bar
sm1u           up   infinite      1   idle quux
main*          up   infinite      3    mix baz[080-081,083]
main*          up   infinite      2  alloc grault,baz082
main*          up   infinite     12   idle baz[061-070],bar,quux

where baz[081-083] is referring to 3 nodes baz081, baz082 and baz083. no if I ssh into any of these nodes I can list all the processes assigned to a specific user by:

ps -A | grep user1

but that would take a lot of time. How can I automate this process:

1. login to one of the nodes
2. run sinfo
3. extract the information and make a list of strings from the last column of sinfo command
4. find all the running processes with a specific user user1 and prints to the terminal

How can I write an script, preferably compatible with Cmder/ConEmu, to automate these steps.

Answer 1

This will try to login to all nodes and print processes of user1:

    for hostList in $(sinfo -h | awk '{print $6}' | sort -u); do
            for host in $(scontrol show hostname $hostList); do
                    echo $host; ssh $host "ps aux | grep user1";
            done;
    done;

However, as you probably do not want to login to offline nodes, extend the sinfo command to:

        for hostList in $(sinfo -h | grep -v down | awk '{print $6}' | sort -u); do

There might be even better ways to accomplish this with better knowledge of sinfo / scontrol but this script should do the job.

However, if this happens a lot you might consider, talking to your administrators as orphaned processes are bad for the cluster performance in general. They should ideally be cleaned by the system automatically on a regular basis.

Answer 2

Based on Noco's answer and further investigation I wrote the perfect script for my own purpose:

#!/bin/bash 
hostList=$(ssh foo "sinfo -h" | awk '{print $6}' | sed 's/,/\n/g' | awk '!seen[$0]++' | sed '/\[/d')
for host in $hostList; do
    echo $host;
    ssh "$1@$host.uni.com" "ps aux | grep $2";
    #ssh "$1@$host.uni.com" "pkill -f $2";
done;

where foo is the main node in my case. This script accept two arguments as sh script arg1 arg2 where arg1 is the user1 as indicated in OP and arg2 is the keyword it will search for. If one wants to remove all the processes assigned to that key word , he/she should uncomment the line with pkill command.

What I observed is that the nodes showing as baz[080-081,083] are all the same in may case so need to search all.