How to run no more than n parallel subshells

Question

I am trying to run subscripts from a main script, but I want to make sure than no more than n subscripts run at the same time.

The following simplified example illustrates.

Each subscript creates a dummy file in RAM (/dev/shm/) with a name made of a unique timestamp, and deletes it once done.

The main script counts the number of dummy files in /dev/shm/ originating from the subscripts, and doesn't (shouldn't) launch a new subscript if 2 or more of them are already running.

However, the main script seems to ignore the while condition, and launches all 5 subscripts at once.

What's wrong with my code?

mainscript.txt

#!/bin/bash
for counter in $(seq 1 5)
do
        while [ $(ls -1 /dev/shm/|grep "script044"|wc -l) -ge 2 ]
        do
                sleep 0
        done

        xterm -e "bash script044.txt" &
done

exit

script044.txt (subscript)

#!/bin/bash

tempfilename="script044_"$(date +%Y%m%d%H%M%S)_${RANDOM}
echo > /dev/shm/${tempfilename}

for counter in $(seq 1 $(shuf -i 10-45 -n 1))
do
        sleep 1
        printf "${counter}\r"
done

rm /dev/shm/${tempfilename}

exit

Answer 1

(Convention - .txt are just plain text files. .sh files are shell script files.).

Your mainscript.txt script has a race condition. Specifically the while loop starts its next iteration before the script044.txt script is able to create the temporary file. In fact the whole loop is iterated through before any of these files get created.

A more robust way to deal with this sort of thing is to forget the temporary files and use the shell builtin wait instead:

#!/bin/bash

pid_count=0
for counter in $(seq 1 5)
do
    xterm -e "bash script044.txt" &
    if (( ++pid_count > 2 )); then
        wait -n
        ((pid_count--))
    fi
done

This increments a counter every time a subprocess is started. If the counter is greater than 3 then we wait for the next subprocess to finish. When wait returns, we then decrement the counter and go around again to start the next xterm.

You can remove all the tempfilename-related lines from the script044.txt - they are no longer needed.

---

As @chepner points out, the required -n option is only available in bash 4.3 or later.

Answer 2

While it won't help if you are aiming for a shell-only solution, GNU parallel provides a sem command that can help with this exact situation.

The following (untested since I don't have your script) should run your job 5 times, but only 2 at a time, waiting to exit at the end:

LIMIT=2
for i in {1..5}; do
    sem -j $LIMIT 'term -e "bash script044.txt"'
done
sem --wait

Answer 3

It looks like you want IPC. Rather than looping over sleep and doing a test each time you do, you can just wait for the child process to tell when its done. This is what pipes are for.

You can have the child processes report to the parent. Open a pipe and share it with them. When they're through, they just need to let the parent know.

sub()(  trap "echo >&9" 0
        sleep 5
)
eval    "exec 9<>"<(echo);i=0
until   [ "$((i+=1))" -gt 5 ]
do      sub & read na <&9
        date +%S:%t"$i"
done

I open it using process substitution. If you can't do this in your shell, you can use instead:

mkfifo pipe; exec 9<>pipe; rm pipe; echo >&9

Now the initial echo puts one line in the pipe - in both cases. That puts you one wait process ahead from the start - which means you'll run two concurrent processes all along. This script uses date to report on the seconds between each sub() call. Here's the output:

34: 1
39: 2
39: 3
44: 4
44: 5

There. Now as you see, every 5 seconds a child dies and when it does it echo's a line to the pipe that read is currently blocking on. As soon as read find's a newline in input, it can quit what it's doing and start over.

You just need to put a trap in your script_044 which instructs the child processes to call out to father when they're through.

Answer 4

A generic way to accomplish this, is to use xargs

printf "%s\n" {1..5} | xargs -P2 -n1 -i xterm -e '/bin/echo Job {}; bash'

printf is only there to fill xargs with values. xargs default replace-str is {} if the -i flag is used. Please read the manpage of xargs to get more detail informations. This should work with the GNU tools and bash as shell.

xargs flags: 
-P     Run up to max-procs processes at a time.[…]
-n     Use  at most max-args arguments per command line.[…]
-i/-I  Replace occurrences of replace-str in the initial-arguments with names read from standard input.[…]

Answer 5

Insert some delay just after launching a new instance of xterm...

        xterm -e "bash script044.txt" &
        sleep 0.1

Answer 6

Using GNU Parallel it looks like this:

seq 1 5 | parallel -j2 -N0 'xterm -e "bash script044.txt"'

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to. It can often replace a for loop.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

Installation

If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

Learn more

See more examples: http://www.gnu.org/software/parallel/man.html

Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel