PHP Cannot get the text from an URL (file_get_content, CURL did not work)

Question

I am trying to get the text from the following url. http://stp.stheadline.com/data/indexNewsMarquee.json

I have tried several methods but none of them worked. I am really desperate right now please help me and thanks in advance.

P.S. I have enabled allow_url_fopen in my ini.php already. P.S. I am using XAMPP v3.3.2 and PHP v5.6.23

the following are the codes that I have tried (and failed)

CURL

$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_URL, $url);
$result = curl_exec($ch);
curl_close($ch);
echo $result;

result: Empty string

file_get_contents

echo file_get_contents($url, true);

result: failed to open stream: HTTP request failed!

readfile

echo readfile($url)

result: output the address itself

See the following link. No need of curl function. stackoverflow.com/questions/19758954/… — Nithyanandhan M
– Nithyanandhan M, Commented Aug 11, 2016 at 9:02

Riad · Accepted Answer · 2016-08-11 09:14:51Z

2

Hi you have to pass header in your curl request, so server can treat its as browser request.

 $url="http://stp.stheadline.com/data/indexNewsMarquee.json";

 $requestHeaders = array(
    "Accept:application/json, text/javascript, */*; q=0.01",
    "Accept-Language:en-US,en;q=0.8",
    "Connection:keep-alive",
    "Host:stp.stheadline.com",
    "Origin:http://stp.stheadline.com",
    "Referer:http://stp.stheadline.com/data/indexNewsMarquee.json",
    "User-Agent:Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36"
 );

 $ch = curl_init();
 curl_setopt($ch, CURLOPT_URL, $url);
 curl_setopt($ch, CURLOPT_HTTPHEADER, $requestHeaders);
 curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
 $output = curl_exec($ch);
 curl_close($ch);
 echo $output;

edited Aug 11, 2016 at 9:14

Riad

3,8585 gold badges30 silver badges39 bronze badges

answered Aug 11, 2016 at 9:07

Haresh Vidja

8,4913 gold badges29 silver badges44 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

yploo Over a year ago

Thanks it works. can you explain a bit what I did wrong (what does it mean to pass the header) I am quite new to this so I would really appreciate it if you can elaborate a little.Thanks in advance

Haresh Vidja Over a year ago

Yeah @Kevin, it seems stp.stheadline.com has implemented security for prevent access unkown requests on server. By including headers in curl request like origin, referer, urger agent.. so that server ensures that request come from browser... it like some tricks to use CURL request as browser request :).. I hope you like my answer and give up vote and mark as correct... thanks in advance

Haresh Vidja Over a year ago

You can refere more from sitepoint.com/using-curl-for-remote-requests

yploo Over a year ago

thanks for your explanation. How can determine if the website has the security implementation. Just curious. Thank you.

Professor Abronsius · Accepted Answer · 2016-08-11 08:58:06Z

2

They are requiring a useragent to be present in the request, so try:

$url='http://stp.stheadline.com/data/indexNewsMarquee.json';
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'banana');
curl_setopt($ch, CURLOPT_URL, $url);
$result = curl_exec($ch);
curl_close($ch);
echo $result;

answered Aug 11, 2016 at 8:58

Professor Abronsius

33.9k5 gold badges36 silver badges49 bronze badges

3 Comments

Andrei Over a year ago

I'm genuinely curios how'd you figure out the need useragent in the request...?

Professor Abronsius Over a year ago

Experience of trying curl requests has shown that on many occasions when a request has failed silently that adding a useragent makes it work ( as it did in this request ) ~ presumably because some bots do not have a useragent and thus do not appear as valid browser traffic.

Andrei Over a year ago

Huh, well. Good to know.

Dharmendra · Accepted Answer · 2016-08-11 09:01:52Z

0

Use Just like this:

 $curl_handle=curl_init();
 curl_setopt($curl_handle,CURLOPT_URL,'Your URL');
 curl_setopt($curl_handle, CURLOPT_CONNECTTIMEOUT, 2);
 curl_setopt($curl_handle, CURLOPT_RETURNTRANSFER, 1);
 curl_setopt($curl_handle, CURLOPT_USERAGENT, 'Your application name');
$query = curl_exec($curl_handle);
curl_close($curl_handle);
print_r($query);

I checked this Code from Your URL.

answered Aug 11, 2016 at 9:01

Dharmendra

12712 bronze badges

3 Comments

Dharmendra Over a year ago

and Leave Section of "Your Application Name" and tried out

yploo Over a year ago

Thanks it works. In your code, there are (CURLOPT_USERAGENT and **CONNECTION_TIMEOUT**(which I dont have). Can you tell me what the code did? Thanks in advance.

Dharmendra Over a year ago

@Kevin 1. CONNECTTIMEOUT : The number of seconds to wait while trying to connect. (0 For infinite). 2. USERAGENT : The contents of the "User-Agent: " header to be used in a HTTP request.

Vatsal Shah · Accepted Answer · 2016-08-11 09:26:19Z

Kevin,

Please try below Code.

header('Content-Type: text/html; charset=utf-8');

$url='http://stp.stheadline.com/data/indexNewsMarquee.json';

$ch = curl_init();
curl_setopt($ch, CURLOPT_HTTPHEADER, array('text/html; charset=utf-8'));
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_ENCODING, "");
curl_setopt($ch, CURLOPT_URL, $url);
$result = curl_exec($ch);
curl_close($ch);
$final_result = json_decode($result);

echo "<pre>";
print_r($final_result);

I have added header('Content-Type: text/html; charset=utf-8'); and curl_setopt($ch, CURLOPT_ENCODING, ""); in your code, and it will work well, also will display correct characters as well.

Rohit Dhiman · Accepted Answer · 2016-08-11 09:43:59Z

Try this out.

$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_URL, 'http://stp.stheadline.com/data/indexNewsMarquee.json');
$result = curl_exec($ch);
curl_close($ch);

echo($result); // Json data

/*$obj = json_decode($result); // To get result in array
print_r($obj);*/

Or

$url = 'http://stp.stheadline.com/data/indexNewsMarquee.json';
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_URL, $url);
$result = curl_exec($ch);
curl_close($ch);

echo($result); // Json data

It's working for me :)

Collectives™ on Stack Overflow

PHP Cannot get the text from an URL (file_get_content, CURL did not work)

CURL

file_get_contents

readfile

5 Answers 5

4 Comments

3 Comments

3 Comments

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

CURL

file_get_contents

readfile

5 Answers 5

4 Comments

3 Comments

3 Comments

Comments

Comments

Linked

Related