Python Web Scraping - Save data in CSV

Question

I am trying to save the data scraped from URLs such as "https://www.holidify.com/places/shimla/mall-road-shimla-sightseeing-3502.html". When saving the data in a csv file only the data from last url from the range gets saved in the csv file. I need that data from all of the URLs gets saved in the csv file.

pages = []
for i in range(1, 10, 1):
    url = "https://www.holidify.com/places/shimla/mall-road-shimla-sightseeing-350" + str(i) + '.html'
    pages.append(url)
    for item in pages:
        page = requests.get(item)
        soup = BeautifulSoup(page.text, 'html.parser')
        Place = list(soup.find(class_="col-md-10 col-xs-10 nopadding"))[1].get_text()
        City = list(soup.find_all(class_="smallerText"))[1].get_text()
        State = list(soup.find_all(class_="smallerText"))[2].get_text()
        Country = list(soup.find_all(class_="smallerText"))[3].get_text()
        About = list(soup.find_all(class_="biggerTextOverview"))[0].get_text()
        more_About = list(soup.find_all(class_="objHeading smallerText"))[0].get_text()
        Weather = soup.find(class_="currentWeather").get_text()
        demo = pd.DataFrame({ "Place": Place, "City": City, "State": State, "Country": Country, "About": About,"More About Places": more_About}, index=[0])
        demo.to_csv('demo.csv', index=False, encoding='utf-8')

Umair Ayub · Accepted Answer · 2019-08-10 08:59:25Z

1

You need to append data into that file

demo.to_csv('demo.csv', index=False, encoding='utf-8', mode = 'a')

answered Aug 10, 2019 at 8:59

Umair Ayub

21.7k14 gold badges82 silver badges154 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

benazir Over a year ago

Hi, thanks for the quick help. Now the all data does get saved in the csv file but each time with the coloumn names being printed with each entry.

furas Over a year ago

@benazir there is option in to_csv to skip headers. You will have to use it in all loops except first. So maybe you should write headers before all loops and then write always without headers. Or put all data in list/dict and save it only once after forloop.

Yash Kanojia · Accepted Answer · 2019-08-10 09:26:36Z

0

As suggested by @Umair append the data into the data frame and place the command demo.to_csv('demo.csv', index=False, encoding='utf-8') outside the loop.

answered Aug 10, 2019 at 9:26

Yash Kanojia

15410 bronze badges

Collectives™ on Stack Overflow

Python Web Scraping - Save data in CSV

2 Answers 2

2 Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Related