Pandas Data frame returning only the first Row of JSON Data?

Question

I'm scraping data using the Twitter API, when I use the print command I can see all the data that i want, specifically as many rows of tweets and dates that I input.

However when I format the data into a pandas data frame/csv it only displays the first row of results. I'm really confused what to do and appreciate all help a lot. thanks :)


#importing key term and date of tweets from twitter archive


client_key = 'code'
client_secret = 'code'

import base64

key_secret = '{}:{}'.format(client_key, client_secret).encode('ascii')
b64_encoded_key = base64.b64encode(key_secret)
b64_encoded_key = b64_encoded_key.decode('ascii')

import requests

base_url = 'https://api.twitter.com/'
auth_url = '{}oauth2/token'.format(base_url)

auth_headers = {
    'Authorization': 'Basic {}'.format(b64_encoded_key),
    'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'
}

auth_data = {
    'grant_type': 'client_credentials'
}

auth_resp = requests.post(auth_url, headers=auth_headers, data=auth_data)
auth_resp.status_code

auth_resp.json().keys()

access_token = auth_resp.json()['access_token']

search_headers = {
    'Authorization': 'Bearer {}'.format(access_token)
}

search_params = {
    'q': 'Key Term',
    'count': 5,
    'start_time' : '2019-1-1',
    'end_time' : '2019-2-1',
    'place.fields' : 'USA',
    'lang' : 'en'
}


search_url = '{}1.1/search/tweets.json'.format(base_url)

search_resp = requests.get(search_url, headers=search_headers, params=search_params)

tweet_data = search_resp.json()

import numpy as np
import pandas as pd


for x in tweet_data['statuses']:

 data = {'Date':[(x['created_at'])],'Text':[(x['text'])]}

df = pd.DataFrame(data)

df.to_csv("Tweet_data.csv")

print(df)

Reuben · Accepted Answer · 2021-06-13 19:19:46Z

1

Hey before your loop define data=[], then inside your loop do data.append({…}).

What you have at the minute is a loop that at every iteration, creates a dictionary and assigns it to a variable called “data”. Overwriting the previous “data” assignment.

Then you are writing a csv with only one “data” row.

Hope that’s helpful!

edited Jun 13, 2021 at 19:19

answered Jun 11, 2021 at 23:11

Reuben

587 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Daniel Over a year ago

thanks, that makes sense, but I'm not too sure how to fix it in execution though, I'm still fairly new when it comes to python

Reuben Over a year ago

Sorry I explained wrongly in my answer I think you will want to make data a list. E.g. before your for loop, data=[] then inside your loop do data.append(<your_dict>)

Collectives™ on Stack Overflow

Pandas Data frame returning only the first Row of JSON Data?

1 Answer 1

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Related