Skip to main content
3 of 4
edits

Bash : parallel curl and variables

I'd like to do this script in bash, so I can understand it and write it in python (that I know much less) later.

I have some files in data. 3, nothing more.

https://www.example.com/data/file27-1.jpg
https://www.example.com/data/file51-2.jpg
https://www.example.com/data/file2576-3.jpg
URL='https://www.example.com/data/file'
i=1 ; j=1
for i in {1..3000} ; do
  curl -I ${URL}file${i}-{j}.jpg
  # here, pipe result into grep,
  ### if !200 ((i++)) (so do nothing) ;
  ### if 200 wget $URL$i$j and ((j++)), and i=i-1 so I can have the same $i at the next loop, for a incremented $j
  " and go on into the the for loop
done

But curling for 3000 links individually takes some times. I'd like to parralelize the curl -I URL in some way, and when I get a 200 response, stop all process requesting as there won't be two files with the same $j value, add 1 to $j, and take everything back to appropriate values $i and $j and go on.

I'm stuck at parallelizing (but found many threads on it), but the part that really blocks me is where a 200 would kill all curl processus, and then resume to the 200 OK $i and $j value.

I hope I've been understandable. I didn't wrote a sample script yet, I'm looking into methods of achieving it.

Thanks