Parallelize for loop in bash

Question

I have the following snippet in my bash script

#!/bin/bash
    for ((i=100; i>=70; i--))
      do
        convert test.png -quality "$i" -sampling-factor 1x1 test_libjpeg_q"$i".jpg
      done

How can i execute the for loop in parallel using all cpu cores.I have seen gnu parallel being used but here i need the output filename in a specific naming scheme as shown above

turn your convert .. line into a separate shell script that parallel can call . Good luck. — shellter
– shellter, Commented Aug 28, 2017 at 17:10
Note that, if performance is your primary concern, and depending on the ratio of the sizes of each image to the total number of images, it may be better to use mogrify and GNU Parallel's -X option because if you use convert, you necessarily have to create a whole new process for every image. So if you have a large number of small images, the overhead is enormous. Say you have 80,000 images, you will be better off with 8 mogrify processes doing 10,000 images each, than 80,000 convert processes doing 1 each. — Mark Setchell
– Mark Setchell, Commented Aug 30, 2017 at 7:41

anubhava · Accepted Answer · 2017-08-30 06:40:17Z

6

You can use parallel like this:

parallel \
'convert test.png -quality {} -sampling-factor 1x1 test_libjpeg_q{}.jpg' ::: {100..70}

edited Aug 30, 2017 at 6:40

answered Aug 28, 2017 at 17:14

anubhava

790k67 gold badges603 silver badges671 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Ole Tange Over a year ago

' and " are unneeded.

anubhava Over a year ago

Yes not needed in this example but a good practice in general.

Ole Tange Over a year ago

Yes, if we are talking shell variables. No if we are talking GNU Parallel substitution strings: echo 'a b' | parallel 'echo "{}"' This is because GNU Parallel quotes the string by default.

Collectives™ on Stack Overflow

Parallelize for loop in bash

1 Answer 1

3 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Related