How to add data to mongoDB in python

Question

I am a python noob (working with it for less than a few hours). I'm trying to read in twitter data and store it in a mongo database, but I am getting the following error:

Traceback (most recent call last):
  File "twit_test.py", line 8, in on_receive
    db.posts.insert(data)
  File "/Library/Python/2.6/site-packages/pymongo-2.0.1-py2.6-macosx-10.6-universal.egg/pymongo/collection.py", line 274, in insert
  File "/Library/Python/2.6/site-packages/pymongo-2.0.1-py2.6-macosx-10.6-universal.egg/pymongo/database.py", line 249, in _fix_incoming
  File "/Library/Python/2.6/site-packages/pymongo-2.0.1-py2.6-macosx-10.6-universal.egg/pymongo/son_manipulator.py", line 73, in transform_incoming
TypeError: 'str' object does not support item assignment
Traceback (most recent call last):
  File "twit_test.py", line 17, in <module>
    conn.perform()

My code is very simple:

import pycurl, json
import pymongo

STREAM_URL = "https://stream.twitter.com/1/statuses/sample.json"
USER = "XXXXXXXX"
PASS = "XXXXXXXX"
def on_tweet(data):
  tweet = json.loads(data)
  db.posts.insert(tweet)

from pymongo import Connection
connection = Connection()
db = connection.test
conn = pycurl.Curl()
conn.setopt(pycurl.USERPWD, "%s:%s" % (USER, PASS))
conn.setopt(pycurl.URL, STREAM_URL)
conn.setopt(pycurl.WRITEFUNCTION, on_tweet)
conn.perform()

I'm sure this is a VERY simple fix, hope you guys can help. Thanks!

moliware · Accepted Answer · 2011-11-05 10:32:26Z

2

On receive you have to buffer the content. When a "\r\n" comes, then you get a tweet and it can be stored in mongodb

def on_tweet(data):
    tweet = json.loads(data)
    db.posts.insert(tweet)


 buffer = ""

 def on_receive(data):
     buffer += data.strip()         
     if (data.endswith("\r\n")):    
         if buffer: 
             on_tweet(buffer)
         buffer = ""

EDIT : I though you were using old streaming api. "on_tweet" function should be enough

answered Nov 5, 2011 at 10:32

moliware

10.4k3 gold badges39 silver badges48 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

WildBill Over a year ago

Sorry, I'm not following. Do I only use the on_tweet function in lieu of the on_receive function? I tried that and all of your code above, but neither worked.....

dcrosta · Accepted Answer · 2011-11-08 04:18:54Z

2

PyMongo's insert method takes a dictionary, not a string. The error you're seeing is where PyMongo attempts to assign an ObjectId for the new record (since it doesn't yet have one) before sending to the database.

I think the error is in your on_receive function. Unless pycurl is converting the JSON for you automatically, it's very likely just giving you a raw string result from twitter's API. You should use the json module to decode the string, then handle the resulting type appropriately -- that is, if it's an array, iterate each item, determine whether it needs to be saved (i.e. whether you already have it in your database), and if not, then issue insert just on those elements which are new.

EDIT: You should also add the safe=True keyword argument to insert. If there is an error that is caught on the server side, you will then get an exception from PyMongo which will help diagnose the problem.

edited Nov 8, 2011 at 4:18

answered Nov 4, 2011 at 12:30

dcrosta

26.4k8 gold badges74 silver badges83 bronze badges

4 Comments

WildBill Over a year ago

I don't understand your post. I'm not using the save function, I'm using the insert item. When i simply do a 'curl' on the command line to get tweets and copy/paste that using the same syntax, it works.

dcrosta Over a year ago

My mistake (a typo) -- but insert also requires a dictionary (i.e. a document) to save, not a string. You'll still need to process and convert the data received from pycurl into the correct format (i.e. a dictionary of the appropriate "shape") before you can save to MongoDB. Not sure what you mean about copy and paste...

WildBill Over a year ago

I added a json.loads function to the callback function. It doesn't error out but it fails to load into the DB as well. Am I missing something else?

WildBill Over a year ago

Updated the code. Also to elaborate on copy/paste - if I do a curl on the command line and copy a tweet and paste it into the interactive MongoDB CLI using a db.posts.insert the tweet loads fine.

WildBill · Accepted Answer · 2011-11-09 00:01:30Z

0

The above edits/current code works. I was incorrectly querying the DB and expecting to see more traffic through the mongo console than I did.

Thanks much to the guys who helped, you got me on teh right track and to the right answer!

answered Nov 9, 2011 at 0:01

WildBill

9,31117 gold badges68 silver badges88 bronze badges

Collectives™ on Stack Overflow

How to add data to mongoDB in python

3 Answers 3

1 Comment

4 Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

4 Comments

Comments

Related