for loop in GNU Parallel

Question

I am trying to download a large number of files from a webpage (which contains only a image, so I can use a simple wget), but want to speed it up using GNU Parallel. Can anyone please help me parallelize this for loop? Thanks.

    for i in `seq 1 1000`
    do
    wget -O "$i.jpg" www.somewebsite.com/webpage
    done

Append a whitespace and & at the end of your wget command. — Cyrus
– Cyrus, Commented Sep 20, 2015 at 12:49
A C-style for loop is better than using seq in bash: for ((i=1; i<=1000; i++)). — chepner
– chepner, Commented Sep 20, 2015 at 13:47

user000001 · Accepted Answer · 2015-09-20 13:31:02Z

4

You could do it like this:

 seq 1 1000 | parallel wget www.somewebsite.com/webpage/{}.jpg

You can also use the -P option to specify the number of jobs you want to run concurrently.

Also you may decide to use curl instead like:

parallel -P 1000 curl -o {}.jpg www.somewebsite.com/webpage/{}.jpg ::: {1..1000}

edited Sep 20, 2015 at 13:31

answered Sep 20, 2015 at 12:51

user000001

33.7k14 gold badges86 silver badges113 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

chepner Over a year ago

There's no reason to bring curl into the answer.

tutuDajuju Over a year ago

Is there a reason for that last bit, what's with the ::: {1..10000}?

Collectives™ on Stack Overflow

for loop in GNU Parallel

1 Answer 1

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Related