You can use GNU make with the --jobs option to run things in parallel but limited to the specified number of jobs. You can tailor that number to something that will not kill your machine.

You can use GNU make with the --jobs option to run things in parallel but limited to the specified number of jobs. You can tailor that number to something that will not kill your machine.

Here's an example Makefile that uses targets a-h (these could be your output files e.g.) and runs a (dummy) set of commands for each target:

all: a b c d e f g h

a b c d e f g h:
    echo $@; sleep 10

N.B. The indentation of the command must be a TAB character. See the GNU make documentation for the details of the syntax of Makefiles.

You can invoke make with make --jobs 4 and get the following output (I used time make --jobs 4 below to show the elapsed time):

echo a; sleep 10
echo b; sleep 10
echo c; sleep 10
echo d; sleep 10
b
a
c
d
echo e; sleep 10
echo f; sleep 10
echo g; sleep 10
e
f
echo h; sleep 10
g
h

real    0m20.009s
user    0m0.010s
sys 0m0.011s

The first four were executed in parallel, then the next four, so the total elapsed timed is 20 seconds.