Trying to get json data from URL using Python

Question

I am learning to get json data from a link and use that data later on. But i am getting error: "RuntimeError: maximum recursion depth exceeded while calling a Python object"

Here is my code:

import json
import requests
from bs4 import BeautifulSoup

url = "http://example.com/category/page=2&YII_CSRF_TOKEN=31eb0a5d28f4dde909d3233b5a0c23bd03348f69&more_products=true"
header = {'x-requested-with': 'XMLHttpRequest'}

mainPage = requests.get(url, headers = header)
xTree = BeautifulSoup(mainPage.content, "lxml")

newDictionary=json.loads(str(xTree))

print (newDictionary)

EDIT: Okay I got the response data from using this slight change, here is the new code:

import json
import requests
from bs4 import BeautifulSoup

url = "http://example.com/category/page=2&YII_CSRF_TOKEN=31eb0a5d28f4dde909d3233b5a0c23bd03348f69&more_products=true"
header = {'x-requested-with': 'XMLHttpRequest'}

mainPage = requests.get(url, headers = header

print (mainPage.json())

Robert Moskal · Accepted Answer · 2016-08-27 22:36:06Z

2

Don't use beautiful soup to process a json http response. Use something like requests:

url = "https://www.daraz.pk/womens-kurtas-shalwar-kameez/?pathInfo=womens-kurtas-shalwar-kameez&page=2&YII_CSRF_TOKEN=31eb0a5d28f4dde909d3233b5a0c23bd03348f69&more_products=true"
header = {'x-requested-with': 'XMLHttpRequest'}
t = requests.get(url, headers=True)
newDictionary=json.loads(t)
print (newDictionary)

The beautiful soup object can't be parsed with json.loads() that way.

If you have HTML data on some of those json keys then you can use beautiful soup to parse those string values individually. If you have a key called content on your json, containing html, you can parse it like so:

BeautifulSoup(newDictionary.content, "lxml")

You may need to experiment with different parsers, if you have fragmentary html.

edited Aug 27, 2016 at 22:36

answered Oct 13, 2015 at 12:23

Robert Moskal

22.7k8 gold badges69 silver badges91 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Mansoor Akram Over a year ago

Please see my edited question, new code returns json data but now i want some tags inside data, How can i do that?

Robert Moskal Over a year ago

Tags? What do you mean by tags in your json?

Mansoor Akram Over a year ago

data is in json format but inside there is html code. I need to retrieve that.

Robert Moskal Over a year ago

Then you need to parse the string data on the json key with beautiful soup and manipulate it the usual way. You should probably post a new question on that.

Mansoor Akram Over a year ago

can you please direct me to some examples of it?

jesterjunk · Accepted Answer · 2015-10-19 07:53:22Z

The following is an example of how to use various JSON data that has been loaded as an object with json.loads().

Working Example — Tested with Python 2.6.9 and 2.7.10 and 3.3.5 and 3.5.0

import json

json_data = '''
{
    "array": [
        1,
        2,
        3
    ],
    "boolean": true,
    "null": null,
    "number": 123,
    "object": {
        "a": "b",
        "c": "d",
        "e": "f"
    },
    "string": "Hello World"
}
'''

data = json.loads(json_data)

list_0 = [
    data['array'][0],
    data['array'][1],
    data['array'][2],
    data['boolean'],
    data['null'],
    data['number'],
    data['object']['a'],
    data['object']['c'],
    data['object']['e'],
    data['string']
]

print('''
array value 0           {0}
array value 1           {1}
array value 2           {2}
boolean value           {3}
null value              {4}
number value            {5}
object value a value    {6}
object value c value    {7}
object value e value    {8}
string value            {9}
'''.format(*list_0))

Output

array value 0           1
array value 1           2
array value 2           3
boolean value           True
null value              None
number value            123
object value a value    b
object value c value    d
object value e value    f
string value            Hello World

Collectives™ on Stack Overflow

Trying to get json data from URL using Python

2 Answers 2

5 Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Related