python requests json from url

Question

I am using python to scrape a url such as in the code blow

import requests
from bs4 import BeautifulSoup
import json

n_index = 10
base_link = 'http://xxx.xxx./getinfo?range=10&district_id=1&index='
for i in range (1,n_index+1):
    link = base_link+str(i)
    r = requests.get(link)
    pid = r.json()
    print (pid)

it's return ten result just like this blow

{'product_info': [{'pid': '1', 'product_type': '2'}]}
{'product_info': [{'pid': '2', 'product_type': '2'}]}
{'product_info': [{'pid': '3', 'product_type': '2'}]}
{'product_info': [{'pid': '4', 'product_type': '2'}]}
{'product_info': [{'pid': '5', 'product_type': '2'}]}
{'product_info': [{'pid': '6', 'product_type': '2'}]}
{'product_info': [{'pid': '7', 'product_type': '2'}]}
{'product_info': [{'pid': '8', 'product_type': '2'}]}
{'product_info': [{'pid': '9', 'product_type': '2'}]}
{'product_info': [{'pid': '10', 'product_type': '2'}]}

and then i want to save the resulting 10 lines into a json file, as presented in the code below:

with open('sylist.json', 'w') as outfile:
    json.dump(r.json(), outfile, indent=4)

but only one result is saved into the json file local, who can help me to resolve,thanks a lot

Use append instead of write ::- with open('sylist.json', 'a') as outfile: — Arvind
– Arvind, Commented Jan 9, 2018 at 5:52

Hrant Davtyan · Accepted Answer · 2018-01-09 06:28:22Z

2

On a typical way, try below way to write result line by line without open/close file at each time.

with open('sylist.json', 'a+') as outfile:
    for i in range (1,n_index+1):
        link = base_link+str(i)
        r = requests.get(link)
        outfile.write("{}\n".format(json.dump(r.json(), outfile, indent=4)))

edited Jan 9, 2018 at 6:28

Hrant Davtyan

5165 silver badges13 bronze badges

answered Jan 9, 2018 at 5:55

Frank AK

1,78117 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Frank AK Over a year ago

@slackware hey, if it worked for you, please accept it:)

Hrant Davtyan · Accepted Answer · 2018-01-09 06:22:26Z

Let me extend Frank's answer a bit. You are sending the request inside the for loop, which means at every iteration of the loop, the value of pid is overwritten. As a result, when you want to dump its content to an output file, pid holds only the contents from the very last iteration/request. I would suggest to apply one of the following to address your issue:

Include writing component inside the for loop (or vice-versa, as suggested in the answer by Frank AK).

Instead of overwriting the content of pid each time, you may append it directly inside the for loop as follows:

my_list = []
for i in range (1,n_index+1):
    link = base_link+str(i)
    r = requests.get(link)
    pid = r.json()
    my_list.append(pid)

with open('sylist.json', 'w') as outfile:
    json.dump(my_list, outfile, indent=4)

Collectives™ on Stack Overflow

python requests json from url

I am using python to scrape a url such as in the code blow

2 Answers 2

1 Comment

Comments

Hot Network Questions

Collectives™ on Stack Overflow

I am using python to scrape a url such as in the code blow

2 Answers 2

1 Comment

Comments

Related