bash script 2000 cURL request

Question

i'm new with bash script so please keep calm with me ^^

I want to write bash script that request 2000 cURL request

is it fast & possible ?

or what should I do for this situation ?

Thanks

EDIT

This is the script I got it from here

#!/bin/bash

url=http://www.***.com/getaccount.php?username=
while read users
do
content=$(curl "{$url}${users}")
echo $users
echo $content >> output.txt
done < users.txt

which users.txt has 2000 username

the question is, is it fast ? because I have to call that script every minute with my crontab .. so it is good for me ? or should I use another language just like Perl or whatever.

before I did 2000 request by crontab but it is very bad idea to add 2000 line to the crontab

so any idea ?

Adam Rosenfield · Accepted Answer · 2013-08-22 01:22:55Z

If all of the URLs you're requesting follow a simple pattern (such as all of the numbered pages from page1.html through page2000.html), then curl itself can easily download them all in one command line:

# Downloads all of page1.html through page2000.html.  Note the quotes to
# protect the URL pattern from shell expansion.
curl --remote-name-all 'http://www.example.com/page[1-2000].html'

See the section labeled "URL" in the manual page for more information on URL patterns.

If you have a lot of URLs which don't follow a numeric pattern, you can put all of the URLs into a file use the -K option of curl to download them all in one go. So, using your example, what you'd want to do is modify your file to convert the usernames into URLs with a prefix of url =. One way to do that is with the sed(1) utility

# Convert list of usernames into a curl options file
sed 's|^\(.*\)$|url = http://www.***.com/getaccount.php?username=\1|' users > curl.config

# Download all of the URLs from the config file
curl --remote-name-all -K curl.config

This will be much faster than downloading individual files in separate commands, because curl can then enable HTTP pipelining within a single process. That way, it sets up a single TCP stream that gets reused for multiple requests, instead of needing to setup a new TCP stream for each request just to tear it down again, which is what would happen if you made each request in a separate process.

Do note, though, that such a large automated download may violate a site's terms of use. You should check a site's robots.txt file before doing such a task, and make sure you're not exceeding their rate limits.

Please check the edit above your reply will be highly appreciated
I did the second code but the question can I add curl --remote-name-all -K curl.config as cron job ? like this * * * * * curl --remote-name-all -K curl.config
@user2104662: Put those commands into a shell script, and then add that shell script to your crontab as a cron job.
Running multiple curl instances in parallel might speed up things if the CPU is idling, and network bandwidth isn't saturated by the first instance of curl, for example 4 instances for 500 urls each.
@Samveen: Doing HTTP requests is highly unlikely to ever be CPU-bound, it's almost always going to be network-bound. If you're accessing multiple different host names, then yes, you could definitely get a speed-up by running the requests in parallel. But if all of your requests are going to the same host name, you may or may not get any speedup from parallel requests, depending on whether the bottleneck is the server or the network pipes.

Jed Daniels · Accepted Answer · 2013-08-22 01:09:56Z

Well, I think you'll need to put down a lot more information to really get a good answer here, but you can make a loop in bash pretty easily:

for i in {1..2000}
do
     echo "This is iteration number $i"
     curl foo
done

The above command will execute each loop sequentially, and all the output will just go to your terminal. You may want to investigate redirecting stdout and stderr, as well as backgrounding the parts you care about.

I highly recommend http://www.tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html and http://www.tldp.org/LDP/abs/html/. Those are my favorite resources for figuring out bash stuff (aside from StackOverflow, of course).

Regarding your question, "is it fast", this depends on your definition of fast. I'm sure the above could be optimized in lots of ways, and I'm even more sure that if you do it in another language it could be much much faster. But it might be fast enough to do whatever it is you are actually trying to do.

Collectives™ on Stack Overflow

bash script 2000 cURL request

2 Answers 2

5 Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Related