6

I'm on a Mac computer accessing by ssh a linux machine to run simulations. I'm currently running multiple commands in parallel (I put them in background with &), when they finish I extract the files of interest then delete all commands output and redo it.

I wondered if it was possible to check using the PID if those tasks are finished in order to automatically extract the wanted files and launch one more time the exact same command. I'm posting a message here to know to get the PID of a command which is executed through a ssh -c, that is not done by me (as I'm in remote linux).

I tried the solutions shown in how-to-get-pid-of-just-started-process in order to get the PID, but neither $! nor ssh -c nor jobs -p give me the correct PID. I'm wondering if it doesn't work because I'm on remote access (the command appearing in htop is a ssh -c ...) or I'm just doing things poorly.

Here is my first bash script :

#!/bin/bash

./createFiles.sh $1 # create the needed $1 folders

for i in $(seq 1 $1)
do
    cd $i
    myCommand &
    cd ..
done

When this one is finished I use :

#!/bin/bash

for i in $(seq 1 $1)
do
    cd $i
    cp output/file.txt ../file_$2_$i.txt # $2 is the number of the run
    cd ..
done

./deleteFiles.sh $2 # delete the needed $1 folders to start anew

And then I loop 5 times those two. And I wanted to know if it's possible to loop automatically 5 times and not having me standing in front of my computer.

If any of you have any idea, it would be great :)

P.S: I hope it was clear, english is not my native language hihi

Improve this question
2
  • 2
    You can use $$ to get the PID from inside your script. You could also use pgrep to find the PID of a given process name from outside your script. Commented Mar 18, 2022 at 14:07
  • I didn't know of pgrep existence, thank you very much :) Commented Mar 18, 2022 at 14:17

2 Answers 2

Reset to default
6

You are building GNU Parallel :)

--transfer --return --cleanup is doing exactly what you are trying to do.

Read chapter 8: https://zenodo.org/record/1146014

Improve this answer
1
  • Oh ! That's exactly what I needed :) Thank you Commented Mar 22, 2022 at 15:09
0

I managed to do it thanks to @mashuptwice answer (i.e : You can use $$ to get the PID from inside your script. You could also use pgrep to find the PID of a given process name from outside your script. – mashuptwice Mar 18 at 14:07)

Then using bash while, -s and sleep I arrived where I wanted :D

For the sake of transparency :

#!/bin/bash

for j in $(seq 1 $2)
do
    ./createFiles.sh $1 # create the needed $1 folders

    for i in $(seq 1 $1)
    do
        cd $i
        myCommand &
        cd ..
    done

    sleep 5 # Wait to be sure that the commands were started

    pgrep -u myUser myCommand > PIDs.txt

    while [ -s PIDs.txt ]
    do
        echo not finished yet
        pgrep -u myUser myCommand > PIDs.txt
        sleep 10
    done
    cat PIDs.txt # Debug print

    ./mvFiles.sh $1 $j # extract the needed files and remove the now useless outputs
    rm PIDs.txt
done
Improve this answer
1
  • 1
    mashuptwice didn't write an answer. You should include the relevant information in your answer, so that it is complete (and continues to make sense in the future when comments get tidied up). Commented Mar 19, 2022 at 8:47

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.