Skip to main content
added 2 characters in body
Source Link
stacko
  • 801
  • 2
  • 8
  • 17

I tried to scrape Google image search result page with curl from Terminal, but it doesn't give me an actual html source that I can view with "View Page Source" in Firefox. I tried both "curl [url]" and "curl -L [url]". Both gave me a short html source that includes "Your client does not have permission to get URL " "from this server". How can I get the html source that I can get in Firefox with a shell script?

Part of the short html I got in Terminal said this.

Please see Google's Terms of Service posted at http://www.google.com/terms_of_service.html

If you believe that you have received this response in error, please report your problem. However, please make sure to take a look at our Terms of Service (http://www.google.com/terms_of_service.html). In your email, please send us the entire code displayed below.

I tried to scrape Google image search result page with curl from Terminal, but it doesn't give me an actual html source that I can view with "View Page Source" in Firefox. I tried both "curl [url]" and "curl -L [url]". Both gave me a short html source that includes "Your client does not have permission to get URL " "from this server". How can I get the html source that I can get in Firefox with a shell script?

I tried to scrape Google image search result page with curl from Terminal, but it doesn't give me an actual html source that I can view with "View Page Source" in Firefox. I tried both "curl [url]" and "curl -L [url]". Both gave me a short html source that includes "Your client does not have permission to get URL " "from this server". How can I get the html source that I can get in Firefox with a shell script?

Part of the short html I got in Terminal said this.

Please see Google's Terms of Service posted at http://www.google.com/terms_of_service.html

If you believe that you have received this response in error, please report your problem. However, please make sure to take a look at our Terms of Service (http://www.google.com/terms_of_service.html). In your email, please send us the entire code displayed below.

Source Link
stacko
  • 801
  • 2
  • 8
  • 17

Curl and Google Image

I tried to scrape Google image search result page with curl from Terminal, but it doesn't give me an actual html source that I can view with "View Page Source" in Firefox. I tried both "curl [url]" and "curl -L [url]". Both gave me a short html source that includes "Your client does not have permission to get URL " "from this server". How can I get the html source that I can get in Firefox with a shell script?