Run subshells in parallel and collect exit codes

Question

I have the following, I thought it ran the subshells in parallel, but it actually seems to run them in series, and I cannot figure out why:

#!/usr/bin/env bash

set -e;
set -m # allow for job control


EXIT_CODE=0;  # exit code of overall script

function handleJobs() {
     for job in `jobs -p`; do
         echo "PID => ${job}"
         CODE=0;
         wait ${job} || CODE=$?
         if [[ "${CODE}" != "0" ]]; then
            echo "At least one process failed with exit code => ${CODE}" ;
            EXIT_CODE=1;
         fi
     done
}

trap 'handleJobs' CHLD


for file in "$HOME/mongodump_dev/cdt_db/"* ; do

  file="$(basename "$file")"

  if [[ "$file" != "system"* && "$file" != "locks"* ]]; then

    mongorestore \
        --db "cdt_dev" \
        --collection "${file%.*}" \
        --host "<my-host>" \
        "$HOME/mongodump_dev/cdt_db/$file"  &

  fi;

done


wait;
echo "exit code => $EXIT_CODE"
exit "$EXIT_CODE"

Anyone know why the subshells are running in series?

I tried doing this instead:

(
        mongorestore \
            --db "cdt_dev" \
            --collection "${file%.*}" \
            --host "<my-host>" \
            "$HOME/mongodump_dev/cdt_db/$file"  &

) &

now they run in parallel, but now the whole script will never exit, and I fear that I am actually not accurately capturing the exit code.

Answer 1

You have a number of "issues" with your script as written. What I did was to copy your script, and for the purposes of testing and evaluation, simplify it by doing thing s like replacing mongorestore with sleep 9999, and by inserting echo trace message that append to a log file. The long and short of it is that it doesn't seem to me that you need any traps; just replace your mongorestore statement with:

( mongorestore ...
  exit_code=$?
  [ $exit_code -ne 0 ] \
  && printf "Process %s for file %s failed with exit code %s\n" \
       $BASHPID  "${file%.*}" $exit_code
) &

BTW, a simple way to verify that all your sub-processes have been spawned in parallel is open another terminal and run pgrep -a mongorestore.

Also, your use of EXIT_CODE at the end of your script seems to me should never have a value in it, so your exit command should fail with some kind of error, which would mean your script would always exit with an error.