How might I execute this nested for loop in parallel?

Question

#!/usr/bin/bash

TARGETS=(
"81.176.235.2"
"81.176.70.2" 
"78.41.109.7" 
)

myIPs=(
"185.164.100.1"
"185.164.100.2"
"185.164.100.3"
"185.164.100.4"
"185.164.100.5"
)

for t in "${TARGETS[@]}"
  do 
    for a in "${myIPs[@]}"
    do 
      echo "${a} ${t} -p 80" >>log 2>&1 &
      echo "${a} ${t} -p 443" >>log 2>&1 &
      wait
    done 
    
done

I want this code to start with echo commands for each IP in TARGETS executing them in parallel. At the same time the script is not meant to proceed with echo commands for more than one address in myIPs simulteously, hence I introduced wait in the internal loop. I want to have pairs of echo (each for the port 80 and 443) executed in parallel for each target in TARGETS. In other words I want to accomplish this (but sadly it does not work):

for t in "${TARGETS[@]}"
do &
  for a in "${myIPs[@]}"
  do 
    echo "${a} ${t} -p 80" >>log 2>&1 &
    echo "${a} ${t} -p 443" >>log 2>&1 &
    wait
  done 

done
wait

Yet, because it would increase my load averages too much, I do not want this: :

for t in "${TARGETS[@]}"
do
  for a in "${myIPs[@]}"
  do 
    echo "${a} ${t} -p 80" >>log 2>&1 &
    echo "${a} ${t} -p 443" >>log 2>&1 &
  done 
done
wait

How might I accomplish my objective?

P.S. This is just a snippet of a more complex script. I wanted isolate the relevant issue, hence the use of echo instead of one of the networking commands.

I was going to comment by suggesting GNU Parallel, but you seem to have already tagged that. Did you try to do it with Parallel, or look at the other Q&A's there about it? (I didn't look the concrete issue too closely, just thought about the choice of tool.) — ilkkachu
– ilkkachu, Commented May 27, 2022 at 12:48
@ilkkachu Yes, I did try it: parallel -j0 'for a in "${myIPs[@]}"; do echo "${a} ${t} -p 80" >>log 2>&1; echo "${a} ${t} -p 443" >>log 2>&1;done' ::: "for t in ${TARGETS[@]}" but it does not work. I don't understand parallel very well. — John Smith
– John Smith, Commented May 27, 2022 at 18:00

glenn jackman · Accepted Answer · 2022-05-27 13:07:08Z

2

I find your question hard to understand: you seem to want both parallel and sequential execution.

Do you want this?

for t in "${TARGETS[@]}"; do 
  (
    for a in "${myIPs[@]}"; do 
      echo "${a} ${t} -p 80" >>log 2>&1 &
      echo "${a} ${t} -p 443" >>log 2>&1 &
      wait
    done 
  ) & 
done

each target's for loop is run in a subshell in the background.

answered May 27, 2022 at 13:07

glenn jackman

88.5k16 gold badges124 silver badges179 bronze badges

Yes, that's what I wanted. The trick with parentheses - that' exactly what I was looking for, I guess you call it command grouping, right?

John Smith
– John Smith

2022-05-27 17:56:13 +00:00
Commented May 27, 2022 at 17:56
Yes that's right.

glenn jackman
– glenn jackman

2022-05-27 19:04:48 +00:00
Commented May 27, 2022 at 19:04

Add a comment |

RudiC · Accepted Answer · 2022-05-27 12:26:34Z

2

If I understand correctly, you should reverse the two loops, and put the wait between the dones. Like

for a in "${myIPs[@]}"
  do    for t in "${TARGETS[@]}"
          do    echo "${a} ${t} -p 80"  2>&1 &
                echo "${a} ${t} -p 443" 2>&1 &
          done 
        wait
  done

For each IP in myIPs, it would send a packet of two ports to each of the three TARGETS, and then wait for those six to finish.

answered May 27, 2022 at 12:26

RudiC

9,0592 gold badges12 silver badges22 bronze badges

Add a comment |

Stack Exchange Network

How might I execute this nested for loop in parallel?

2 Answers 2

You must log in to answer this question.

Hot Network Questions

How might I execute this nested for loop in parallel?

2 Answers 2

You must log in to answer this question.

Related

Hot Network Questions