Parsing through a JSON file with Python 2.x

Question

I'm currently trying to parse through a text file containing a number of Facebook chat fragments. The fragments are stored as below:-

{"t":"msg","c":"p_100002239013747","s":14,"ms":[{"msg":{"text":"2what is the best restauran
t in hong kong? ","time":1303115825598,"clientTime":1303115824391,"msgID":"1862585188"},"from":10000
2239013747,"to":635527479,"from_name":"David Robinson","from_first_name":"David","from_gender":1,"to_name":"Jason Yeung","to_first_name":"Jason","to_gender":2,"type":"msg"}]}

I've tried a number of ways to parse / open the JSON file but to no avail. Here is what I've tried thusfar:-

import json

data = []
with open("C:\\Users\\Me\\Desktop\\facebookchat.txt", 'r') as json_string:
    for line in json_string:
        data.append(json.loads(line)) 

error:

Traceback (most recent call last):
  File "C:/Users/Amy/Desktop/facebookparser.py", line 6, in <module>
    data.append(json.loads(line))
  File "C:\Program Files\Python27\lib\json\__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "C:\Program Files\Python27\lib\json\decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Program Files\Python27\lib\json\decoder.py", line 382, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Invalid control character at: line 1 column 91 (char 91)

and also:

import json

with open("C:\\Users\\Me\\Desktop\\facebookchat.txt", 'r') as json_file:
    data = json.load(json_file)

... but I get exactly the same error as above.

Any suggestions? I've searched previous posts on here and tried the alternative solutions but to no avail. I'm aware I need to treat it as a dictionary file with for example, 'time' being a key and '1303115825598' being the respective time value but if I can't even process the json file into memory, there's no way I can parse it.

Where am I going wrong? Thanks

Answer 1

Your data contains newlines where JSON would not allow these. You'll have to stitch the lines back together again:

data = []
with open("C:\\Users\\Me\\Desktop\\facebookchat.txt", 'r') as json_string:
    partial = ''
    for line in json_string:
        partial += line.rstrip('\n')
        try:
            data.append(json.loads(partial))
            partial = ''
        except ValueError:
            continue  # Not yet a complete JSON value

The code collects lines into partial, but minus the newline, and tries to decode the JSON. If that succeeds, partial is set to the empty string again to process the next entry. If it fails, we loop to the next line to append, until there is a complete JSON value to decode.