Get content of page using PHP cURL

Question

I want to use PHP's cURL to visit a page on an external site, and get some the whole html content of the page.

When i visit the site, it will redirect me to another page on the same site. Also, i have to set the useragent, i want a useragent for PC windows7 chrome and iPhone 4s. This is what i got so far:

$ch = curl_init ($url);
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_AUTOREFERER , true)
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
$kl = curl_exec ($ch);
curl_close($ch);
echo $kl;

Notice:
I will probably run into more errors.

@DevZer0 these libs had been designed for PHP4. in PHP5 there is DOMXPath or simple_xml_element->xpath() — hek2mgl
– hek2mgl, Commented Jul 22, 2013 at 1:49
you can get html result, but if you want to manipulate the html, get some items inside it, you should write more code to analyze it. "PHP Simple HTML DOM Parser" it is helpful.And if there is a redirection, you should use curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); to follow it — TroyCheng
– TroyCheng, Commented Jul 22, 2013 at 2:32

Baba · Accepted Answer · 2013-08-15 08:22:38Z

7

You might also need to consider urls with https

$cookie = tmpfile();
$userAgent = 'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31' ;

$ch = curl_init($url);

$options = array(
    CURLOPT_CONNECTTIMEOUT => 20 , 
    CURLOPT_USERAGENT => $userAgent,
    CURLOPT_AUTOREFERER => true,
    CURLOPT_FOLLOWLOCATION => true,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_COOKIEFILE => $cookie,
    CURLOPT_COOKIEJAR => $cookie ,
    CURLOPT_SSL_VERIFYPEER => 0 ,
    CURLOPT_SSL_VERIFYHOST => 0
);

curl_setopt_array($ch, $options);
$kl = curl_exec($ch);
curl_close($ch);
echo $kl;

answered Aug 15, 2013 at 8:22

Baba

95.3k29 gold badges172 silver badges222 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

131 Over a year ago

Thanks, it also did it, and +1 for putting it into array :)

Wrikken · Accepted Answer · 2013-08-12 20:15:13Z

4

+50

So:

Search the proper UserAgent strings on the 'net.
Enable CURLOPT_FOLLOWLOCATION as @TroyCheng indicated
Enable the CURLOPT_COOKIEFILE & CURLOPT_COOKIEJAR.

answered Aug 12, 2013 at 20:15

Wrikken

70.8k8 gold badges99 silver badges136 bronze badges

1 Comment

131 Over a year ago

thanks. it did it. sadly i can't give you the bounty until 23 hours have past :/

j0k · Accepted Answer · 2013-08-15 08:39:37Z

Why don't you use a library like Buzz ?

$request = new Buzz\Message\Request('GET', '/', 'http://google.com');
$response = new Buzz\Message\Response();

$client = new Buzz\Client\Curl();
// do not check https validity
$client->setVerifyPeer(false);
// define your user agent
$client->setOption('CURLOPT_USERAGENT', $userAgent);
$client->setOption('CURLOPT_COOKIEFILE', true);
$client->setOption('CURLOPT_COOKIEJAR', true);
$client->send($request, $response);

if ($response->isOk())
{
  echo $response->getContent();

  // or if you want the dom
  echo $response->toDomDocument();
}

Collectives™ on Stack Overflow

Get content of page using PHP cURL

3 Answers 3

1 Comment

1 Comment

Comments

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

1 Comment

Comments

Related