Bash script for running parallel commands using input from a file

Question

I am trying to make a shell script which reads a configuration file and executes commands for each line in parallel.

For example, I have IPs.cfg, which can contain a variable amount of IPs. It could be one or several

IPs.cfg

145.x.x.x
176.x.x.x
192.x.x.x

I want to read the file and then execute a command for each line at the same time... for instance :

scp test.iso root@$IP1:/tmp &
scp test.iso root@$IP2:/tmp &
scp test.iso root@$IP3:/tmp &
wait

The way I'm thinking this is that I store the IPs into an array

IFS=$'\n' read -d '' -r -a array < IPs.cfg

Then I extract the number of lines from the file and decrease it by 1 since the array starts at 0.

NUMLINES=`cat IPs.cfg | wc -l`
NUMLINES=$((NUMLINES-1))

Now I want to execute the commands all at the same time. It's a variable number of parameters so I can't just manually use scp test.iso root@${array[0]}:/tmp & scp test.iso root@${array[1]}:/tmp & so on. I could use a while loop, but that would mean doing the commands one at a time. I'm also thinking about using recursion, but I have never done that in a bash script.

It might be a silly question, but what are my options here?

unconditional · Accepted Answer · 2016-03-18 13:59:30Z

2

You could make use of GNU parallel

edited Mar 18, 2016 at 13:59

answered Mar 18, 2016 at 13:41

unconditional

7,7541 gold badge19 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

hek2mgl · Accepted Answer · 2016-03-18 13:39:47Z

1

The loop should look like this:

while read -r ip ; do
   scp test.iso "root@$ip:/tmp" &
done < IPs.conf
wait

answered Mar 18, 2016 at 13:39

hek2mgl

159k31 gold badges263 silver badges279 bronze badges

1 Comment

robertpas Over a year ago

I just tested this out. It works flawlessly. I did not even know that you can use & like that. Thank you very much for the quick answer.

user2021201 · Accepted Answer · 2016-03-18 22:53:03Z

0

with this trick, you can control the number of simultaneous processes running at the same:

cat IPs.conf | xargs -n1 -I{} -P10 scp test.iso "root@{}:/tmp"

Check the -p10 which means to use 10 processes at the same time.

answered Mar 18, 2016 at 22:53

user2021201

3903 silver badges10 bronze badges

Collectives™ on Stack Overflow

Bash script for running parallel commands using input from a file

3 Answers 3

Comments

1 Comment

Comments

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

Comments

Related