JSON pretty printing isn't working in Python

Question

As far as I can tell, this code should work.

import json
with open('path', 'r') as infile:
    data = json.load(infile)
    print(json.dumps(data, indent=4))

But it doesn't. I get this:

"b'{\"Markets\":[{\"ID\":2461,\"Name\":\"Who will be elected German chancellor in 2017?\",\"ShortName\":\"Who will be elected German chancellor in 2017?\",\"TickerSymbol\":\"CHANCELLOR.GERMANY.2017\",\"Image\":\"https://az620379.vo.msecnd.net/images/Markets/5914fd50-9a60-46d1-ac5f-b36c2c4c9832.png\",\"URL\":\"https://www.predictit.org/Market/2461/Who-will-be-elected-German-chancellor-in-2017\",\"Contracts\":[{\"ID\":3613,\"DateEnd\":\"N/A\",\"Image\":\"https://az620379.vo.msecnd.net/images/Contracts/small_494e673d-12cc-40c5-ab6a-2976c582303f.png\" ...

However, if I go to http://jsonprettyprint.com/ and give it the same data, I get pretty indentation. So the problem's with the code, not the data (though I don't understand why there are so many backslashes).

What am I doing wrong?

Edit: Sample data

"b'{\"ID\":2461,\"Name\":\"Who will be elected German chancellor in 2017?\",\"ShortName\":\"Who will be elected German chancellor in 2017?\",\"TickerSymbol\":\"CHANCELLOR.GERMANY.2017\",\"Image\":\"https://az620379.vo.msecnd.net/images/Markets/5914fd50-9a60-46d1-ac5f-b36c2c4c9832.png\",\"URL\":\"https://www.predictit.org/Market/2461/Who-will-be-elected-German-chancellor-in-2017\",\"Contracts\":[{\"ID\":3613,\"DateEnd\":\"N/A\",\"Image\":\"https://az620379.vo.msecnd.net/images/Contracts/small_494e673d-12cc-40c5-ab6a-2976c582303f.png\",\"URL\":\"https://www.predictit.org/Contract/3613/Will-Angela-Merkel-be-elected-German-chancellor-in-2017\",\"Name\":\"Angela Merkel\",\"LongName\":\"Will Angela Merkel be elected German chancellor in 2017?\",\"ShortName\":\"Merkel\",\"TickerSymbol\":\"MERKEL.CHANCELLOR.GERMANY.2017\",\"Status\":\"Open\",\"LastTradePrice\":0.10,\"BestBuyYesCost\":0.10,\"BestBuyNoCost\":0.93,\"BestSellYesCost\":0.07,\"BestSellNoCost\":0.90,\"LastClosePrice\":0.10},{\"ID\":3614,\"DateEnd\":\"N/A\",\"Image\":\"https://az620379.vo.msecnd.net/images/Contracts/small_d87c6a74-372d-40b5-95d9-2f994738acf5.png\",\"URL\":\"https://www.predictit.org/Contract/3614/Will-Sigmar-Gabriel-be-elected-German-chancellor-in-2017\",\"Name\":\"Sigmar Gabriel\",\"LongName\":\"Will Sigmar Gabriel be elected German chancellor in 2017?\",\"ShortName\":\"Gabriel\",\"TickerSymbol\":\"GABRIEL.CHANCELLOR.GERMANY.2017\",\"Status\":\"Open\",\"LastTradePrice\":0.01,\"BestBuyYesCost\":0.01,\"BestBuyNoCost\":null,\"BestSellYesCost\":null,\"BestSellNoCost\":0.99,\"LastClosePrice\":0.01},{\"ID\":3615,\"DateEnd\":\"N/A\",\"Image\":\"https://az620379.vo.msecnd.net/images/Contracts/small_be043f00-72fb-479c-b5ae-37e51154935f.png\",\"URL\":\"https://www.predictit.org/Contract/3615/Will-Frank-Walter-Steinmeier-be-elected-German-chancellor-in-2017\",\"Name\":\"Frank-Walter Steinmeier\",\"LongName\":\"Will Frank-Walter Steinmeier be elected German chancellor in 2017?\",\"ShortName\":\"Steinmeier\",\"TickerSymbol\":\"STEINMEIER.CHANCELLOR.GERMANY.2017\",\"Status\":\"Open\",\"LastTradePrice\":0.01,\"BestBuyYesCost\":0.01,\"BestBuyNoCost\":null,\"BestSellYesCost\":null,\"BestSellNoCost\":0.99,\"LastClosePrice\":0.01},{\"ID\":3616,\"DateEnd\":\"N/A\",\"Image\":\"https://az620379.vo.msecnd.net/images/Contracts/small_9d6f499c-245d-4cba-a1b1-537d1a41408d.png\",\"URL\":\"https://www.predictit.org/Contract/3616/Will-Martin-Schulz-be-elected-German-chancellor-in-2017\",\"Name\":\"Martin Schulz\",\"LongName\":\"Will Martin Schulz be elected German chancellor in 2017?\",\"ShortName\":\"Schulz\",\"TickerSymbol\":\"SCHULZ.CHANCELLOR.GERMANY.2017\",\"Status\":\"Open\",\"LastTradePrice\":0.01,\"BestBuyYesCost\":0.01,\"BestBuyNoCost\":null,\"BestSellYesCost\":null,\"BestSellNoCost\":0.99,\"LastClosePrice\":0.01},{\"ID\":3617,\"DateEnd\":\"N/A\",\"Image\":\"https://az620379.vo.msecnd.net/images/Contracts/small_a8e62e71-b5f3-4050-8340-c41204d58e30.png\",\"URL\":\"https://www.predictit.org/Contract/3617/Will-Wolfgang-Sch%C3%A4uble-be-elected-German-chancellor-in-2017\",\"Name\":\"Wolfgang Sch\\xc3\\xa4uble\",\"LongName\":\"Will Wolfgang Sch\\xc3\\xa4uble be elected German chancellor in 2017?\",\"ShortName\":\"Sch\\xc3\\xa4uble\",\"TickerSymbol\":\"SCHAUBLE.CHANCELLOR.GERMANY.2017\",\"Status\":\"Open\",\"LastTradePrice\":0.01,\"BestBuyYesCost\":0.01,\"BestBuyNoCost\":null,\"BestSellYesCost\":null,\"BestSellNoCost\":0.99,\"LastClosePrice\":0.01},{\"ID\":3618,\"DateEnd\":\"N/A\",\"Image\":\"https://az620379.vo.msecnd.net/images/Contracts/small_a41f6020-3a21-4679-ab04-1ff2c4e90e14.png\",\"URL\":\"https://www.predictit.org/Contract/3618/Will-Frauke-Petry-be-elected-German-chancellor-in-2017\",\"Name\":\"Frauke Petry\",\"LongName\":\"Will Frauke Petry be elected German chancellor in 2017?\",\"ShortName\":\"Petry\",\"TickerSymbol\":\"PETRY.CHANCELLOR.GERMANY.2017\",\"Status\":\"Open\",\"LastTradePrice\":0.01,\"BestBuyYesCost\":0.01,\"BestBuyNoCost\":null,\"BestSellYesCost\":null,\"BestSellNoCost\":0.99,\"LastClosePrice\":0.01},{\"ID\":3619,\"DateEnd\":\"N/A\",\"Image\":\"https://az620379.vo.msecnd.net/images/Contracts/small_18819f42-5bba-4640-96fb-e62ce4879cd0.png\",\"URL\":\"https://www.predictit.org/Contract/3619/Will-Thomas-de-Maizi%C3%A8re-be-elected-German-chancellor-in-2017\",\"Name\":\"Thomas de Maizi\\xc3\\xa8re\",\"LongName\":\"Will Thomas de Maizi\\xc3\\xa8re be elected German chancellor in 2017?\",\"ShortName\":\"de Maizi\\xc3\\xa8re\",\"TickerSymbol\":\"DEMAIZIERE.CHANCELLOR.GERMANY.2017\",\"Status\":\"Open\",\"LastTradePrice\":0.01,\"BestBuyYesCost\":0.01,\"BestBuyNoCost\":null,\"BestSellYesCost\":null,\"BestSellNoCost\":0.99,\"LastClosePrice\":0.01},{\"ID\":3620,\"DateEnd\":\"N/A\",\"Image\":\"https://az620379.vo.msecnd.net/images/Contracts/small_deffbe18-669e-45c6-9bb5-62ab61363d49.png\",\"URL\":\"https://www.predictit.org/Contract/3620/Will-Ursula-Von-der-Leyen-be-elected-German-chancellor-in-2017\",\"Name\":\"Ursula Von der Leyen\",\"LongName\":\"Will Ursula Von der Leyen be elected German chancellor in 2017?\",\"ShortName\":\"Von der Leyen\",\"TickerSymbol\":\"VONDERLEYEN.CHANCELLOR.GERMANY.2017\",\"Status\":\"Open\",\"LastTradePrice\":0.01,\"BestBuyYesCost\":0.01,\"BestBuyNoCost\":null,\"BestSellYesCost\":null,\"BestSellNoCost\":0.99,\"LastClosePrice\":0.01},{\"ID\":3621,\"DateEnd\":\"N/A\",\"Image\":\"https://az620379.vo.msecnd.net/images/Contracts/small_0b630af8-c76c-427c-91af-50a2e5f37bc4.png\",\"URL\":\"https://www.predictit.org/Contract/3621/Will-Christian-Lindner-be-elected-German-chancellor-in-2017\",\"Name\":\"Christian Lindner\",\"LongName\":\"Will Christian Lindner be elected German chancellor in 2017?\",\"ShortName\":\"Lindner\",\"TickerSymbol\":\"LINDNER.CHANCELLOR.GERMANY.2017\",\"Status\":\"Open\",\"LastTradePrice\":0.01,\"BestBuyYesCost\":0.01,\"BestBuyNoCost\":null,\"BestSellYesCost\":null,\"BestSellNoCost\":0.99,\"LastClosePrice\":0.01},{\"ID\":3622,\"DateEnd\":\"N/A\",\"Image\":\"https://az620379.vo.msecnd.net/images/Contracts/small_e6aaf4d2-00c7-437c-82c0-d29336d75ad7.png\",\"URL\":\"https://www.predictit.org/Contract/3622/Will-Horst-Seehofer-be-elected-German-chancellor-in-2017\",\"Name\":\"Horst Seehofer\",\"LongName\":\"Will Horst Seehofer be elected German chancellor in 2017?\",\"ShortName\":\"Seehofer\",\"TickerSymbol\":\"SEEHOFER.CHANCELLOR.GERMANY.2017\",\"Status\":\"Open\",\"LastTradePrice\":0.01,\"BestBuyYesCost\":0.01,\"BestBuyNoCost\":null,\"BestSellYesCost\":null,\"BestSellNoCost\":0.99,\"LastClosePrice\":0.01},{\"ID\":3623,\"DateEnd\":\"N/A\",\"Image\":\"https://az620379.vo.msecnd.net/images/Contracts/small_59a0cbcf-8bbf-4f39-ab30-175d9213732f.png\",\"URL\":\"https://www.predictit.org/Contract/3623/Will-Cem-%C3%96zdemir-be-elected-German-chancellor-in-2017\",\"Name\":\"Cem \\xc3\\x96zdemir\",\"LongName\":\"Will Cem \\xc3\\x96zdemir be elected German chancellor in 2017?\",\"ShortName\":\"\\xc3\\x96zdemir\",\"TickerSymbol\":\"OZDEMIR.CHANCELLOR.GERMANY.2017\",\"Status\":\"Open\",\"LastTradePrice\":0.01,\"BestBuyYesCost\":0.01,\"BestBuyNoCost\":null,\"BestSellYesCost\":null,\"BestSellNoCost\":0.99,\"LastClosePrice\":0.01},{\"ID\":3624,\"DateEnd\":\"N/A\",\"Image\":\"https://az620379.vo.msecnd.net/images/Contracts/small_076c33fc-39fc-414f-abf2-cc7dfa2e4dad.png\",\"URL\":\"https://www.predictit.org/Contract/3624/Will-Katja-Kipping-be-elected-German-chancellor-in-2017\",\"Name\":\"Katja Kipping\",\"LongName\":\"Will Katja Kipping be elected German chancellor in 2017?\",\"ShortName\":\"Kipping\",\"TickerSymbol\":\"KIPPING.CHANCELLOR.GERMANY.2017\",\"Status\":\"Open\",\"LastTradePrice\":0.01,\"BestBuyYesCost\":0.01,\"BestBuyNoCost\":null,\"BestSellYesCost\":null,\"BestSellNoCost\":0.99,\"LastClosePrice\":0.01},{\"ID\":3625,\"DateEnd\":\"N/A\",\"Image\":\"https://az620379.vo.msecnd.net/images/Contracts/small_84a1659d-63fd-4169-a5e7-49762df694e2.png\",\"URL\":\"https://www.predictit.org/Contract/3625/Will-Hannelore-Kraft-be-elected-German-chancellor-in-2017\",\"Name\":\"Hannelore Kraft\",\"LongName\":\"Will Hannelore Kraft be elected German chancellor in 2017?\",\"ShortName\":\"Kraft\",\"TickerSymbol\":\"KRAFT.CHANCELLOR.GERMANY.2017\",\"Status\":\"Open\",\"LastTradePrice\":0.01,\"BestBuyYesCost\":0.01,\"BestBuyNoCost\":null,\"BestSellYesCost\":null,\"BestSellNoCost\":0.99,\"LastClosePrice\":0.01},{\"ID\":3626,\"DateEnd\":\"N/A\",\"Image\":\"https://az620379.vo.msecnd.net/images/Contracts/small_80d9880b-1881-4079-9935-c0a5691876e9.png\",\"URL\":\"https://www.predictit.org/Contract/3626/Will-Peter-Altmaier-be-elected-German-chancellor-in-2017\",\"Name\":\"Peter Altmaier\",\"LongName\":\"Will Peter Altmaier be elected German chancellor in 2017?\",\"ShortName\":\"Altmaier\",\"TickerSymbol\":\"ALTMAIER.CHANCELLOR.GERMANY.2017\",\"Status\":\"Open\",\"LastTradePrice\":0.01,\"BestBuyYesCost\":0.01,\"BestBuyNoCost\":null,\"BestSellYesCost\":null,\"BestSellNoCost\":0.99,\"LastClosePrice\":0.01},{\"ID\":3627,\"DateEnd\":\"N/A\",\"Image\":\"https://az620379.vo.msecnd.net/images/Contracts/small_cb11286d-c0b9-41fc-a4b0-3f52dccbd9bf.png\",\"URL\":\"https://www.predictit.org/Contract/3627/Will-Olaf-Scholz-be-elected-German-chancellor-in-2017\",\"Name\":\"Olaf Scholz\",\"LongName\":\"Will Olaf Scholz be elected German chancellor in 2017?\",\"ShortName\":\"Scholz\",\"TickerSymbol\":\"SCHOLZ.CHANCELLOR.GERMANY.2017\",\"Status\":\"Open\",\"LastTradePrice\":0.01,\"BestBuyYesCost\":0.01,\"BestBuyNoCost\":null,\"BestSellYesCost\":null,\"BestSellNoCost\":0.99,\"LastClosePrice\":0.01},{\"ID\":4866,\"DateEnd\":\"N/A\",\"Image\":\"https://az620379.vo.msecnd.net/images/Contracts/small_72649d7f-0d79-4d34-a061-3920669d69c3.png\",\"URL\":\"https://www.predictit.org/Contract/4866/Will-Jens-Spahn-be-elected-German-chancellor-in-2017\",\"Name\":\"Jens Spahn\",\"LongName\":\"Will Jens Spahn be elected German chancellor in 2017?\",\"ShortName\":\"Spahn\",\"TickerSymbol\":\"SPAHN.CHANCELLOR.GERMANY.2017\",\"Status\":\"Open\",\"LastTradePrice\":0.01,\"BestBuyYesCost\":0.01,\"BestBuyNoCost\":null,\"BestSellYesCost\":null,\"BestSellNoCost\":0.99,\"LastClosePrice\":0.01},{\"ID\":4885,\"DateEnd\":\"N/A\",\"Image\":\"https://az620379.vo.msecnd.net/images/Contracts/small_86dc58ff-2801-4174-acb6-3fea5f5b801f.png\",\"URL\":\"https://www.predictit.org/Contract/4885/Will-Leif-Erik-Holm-be-elected-German-chancellor-in-2017\",\"Name\":\"Leif-Erik Holm\",\"LongName\":\"Will Leif-Erik Holm be elected German chancellor in 2017?\",\"ShortName\":\"Holm\",\"TickerSymbol\":\"HOLM.CHANCELLOR.GERMANY.2017\",\"Status\":\"Open\",\"LastTradePrice\":0.01,\"BestBuyYesCost\":0.01,\"BestBuyNoCost\":null,\"BestSellYesCost\":null,\"BestSellNoCost\":0.99,\"LastClosePrice\":0.01},{\"ID\":4920,\"DateEnd\":\"N/A\",\"Image\":\"https://az620379.vo.msecnd.net/images/Contracts/small_bc912c09-4d05-4d0a-9bba-d2dda6621b5d.png\",\"URL\":\"https://www.predictit.org/Contract/4920/Will-Karl-Theodor-zu-Guttenberg-be-elected-German-chancellor-in-2017\",\"Name\":\"Karl-Theodor zu Guttenberg\",\"LongName\":\"Will Karl-Theodor zu Guttenberg be elected German chancellor in 2017?\",\"ShortName\":\"Guttenberg\",\"TickerSymbol\":\"GUTTENBERG.CHANCELLOR.GERMANY.2017\",\"Status\":\"Open\",\"LastTradePrice\":0.01,\"BestBuyYesCost\":0.02,\"BestBuyNoCost\":null,\"BestSellYesCost\":null,\"BestSellNoCost\":0.98,\"LastClosePrice\":0.01},{\"ID\":7974,\"DateEnd\":\"N/A\",\"Image\":\"https://az620379.vo.msecnd.net/images/Contracts/small_a2440597-a99e-4ab3-96cc-0c821bb978a0.png\",\"URL\":\"https://www.predictit.org/Contract/7974/Will-Alexander-Gauland-be-elected-German-chancellor-in-2017\",\"Name\":\"Alexander Gauland\",\"LongName\":\"Will Alexander Gauland be elected German chancellor in 2017?\",\"ShortName\":\"Gauland\",\"TickerSymbol\":\"GAUL.CHANCELLOR.GERMANY.2017\",\"Status\":\"Open\",\"LastTradePrice\":0.01,\"BestBuyYesCost\":0.01,\"BestBuyNoCost\":null,\"BestSellYesCost\":null,\"BestSellNoCost\":0.99,\"LastClosePrice\":0.01},{\"ID\":7975,\"DateEnd\":\"N/A\",\"Image\":\"https://az620379.vo.msecnd.net/images/Contracts/small_9bec806a-8b80-47e2-bd03-c279a2b6b799.png\",\"URL\":\"https://www.predictit.org/Contract/7975/Will-Alice-Weidel-be-elected-German-chancellor-in-2017\",\"Name\":\"Alice Weidel\",\"LongName\":\"Will Alice Weidel be elected German chancellor in 2017?\",\"ShortName\":\"Weidel\",\"TickerSymbol\":\"WEID.CHANCELLOR.GERMANY.2017\",\"Status\":\"Open\",\"LastTradePrice\":0.01,\"BestBuyYesCost\":0.01,\"BestBuyNoCost\":null,\"BestSellYesCost\":null,\"BestSellNoCost\":0.99,\"LastClosePrice\":0.01}],\"TimeStamp\":\"2017-12-09T01:44:29.7878389\",\"Status\":\"Open\"}'"

Edit 2: The admittedly messy code to produce the json file.

from urllib.request import Request, urlopen

q = Request('https://www.predictit.org/api/marketdata/all/')
q.add_header('Accept', 'application/json')

import json
import datetime

def saveData(data):
    now = datetime.datetime.now()
    path = 'json/PI ' + str(now) + '.json'
    path = path.replace(':','-') # No colons in file name
    with open(path, 'w') as outfile:
        json.dump(str(data), outfile, ensure_ascii=False)

import time
starttime=time.time()
while True:
    data = urlopen(q).read()
    saveData(data)
    time.sleep(60.0 - ((time.time() - starttime) % 60.0))

Answer 1

How did you produce this file? It's not a JSON object, it's a JSON string containing a Python repr containing a string of your original JSON. Or something like that. Which you can tell because of all the extra backslashes and the leading b' character.

This works:

import json
from ast import literal_eval

file_data = open('test_file.json', 'r').read()
python_repr = json.loads(file_data)
json_string = literal_eval(python_repr)
actual_dict = json.loads(json_string.decode('utf-8'))
print(json.dumps(actual_dict, indent=4)) # finally.

But how in tarnation did you manage this in the first place?

You can write it back as simple JSON instead of multi-encoded wtf too:

with open('output.json', 'wb') as out_file:
    json.dump(actual_dict, out_file, indent=4)

EDIT for file write:

When you read the data from the website, it's a string containing the JSON object. You need to convert that into a Python dict first, then write it out. Without calling str on it again. That's how you're ending up with the triple-encoded mess. Using your code from above:

data = urlopen(q).read()
json_data = json.loads(data.decode('utf-8')) # DECODE THE STRING 
with open('output.json', 'w') as out:
    json.dump(json_data, out, indent=4) # DO NOT CALL STR ON IT AGAIN

Once you've written it cleanly, you can read it easily too:

json_loaded = json.load(open('output.json', 'r'))

Which will load it up as a python dict.

I'd suggest learning a bit more about the difference between your data itself and various representations of that data.

EDIT2: Also note that if you just want to save the data, you can simply write what you're getting from the website directly into a file, without bothering with the json.load/dump roundtrip.

data = urlopen(q).read()
with open('output.json', 'w') as out:
    out.write(data)