Error in json.loads() for data that has base64 decoding applied

Question

I am trying to use json.loads() in python

I get the error:

the JSON object must be str, not 'bytes'

a = {'h': '123', 'w': '12345', 'data': "b'eyJod2lkIjpwomfcwpvepovnepovqrepniLLKJAMSNDMSNDMAWEFMOEDAad='"}


a.update(json.loads(base64.b64decode(a['data'])))

Here the 'data' portion of a was being loaded in as a json dump with b64encoding.

'data':base64.b64encode(json.dumps(test).encode()); where test = some string eg('epovqrepniLLKJAMSNDMSNDMAWEFMOEDAad=')

I have tried using:

a.update(json.loads(base64.b64decode(a['data']).decode('utf-8')))

Giving me a 'utf-8' codec can't decode bytes in position: invalid continuation byte

I have also tried using decodebytes instead of b64decode to no avail.

I'd really appreciate any help!

You don't need to json.loads(data=a) as it is already a python dict, you just need to output this dict directly. And outputing json string from a dict is json.dumps()'s job — min2bro
– min2bro, Commented Jun 5, 2018 at 7:16
The data value looks pretty corrupted. Do you know what "b'eyJ... was before it was encoded with JSON and B64? By removing the outer quotes and applying b64decode to the byte string b'eyJ...', I get something that looks like JSON in the beginning (b'{"hwid":), but the remainder is clearly not valid JSON. — lenz
– lenz, Commented Jun 5, 2018 at 19:24
@lenz you are correct, these were dummy values. The actual encoded data was valid. I need to be able to remove the outer quotes in the dict. I am however, unable to use json,loads on the dict even though I used b64decode — LemonSnippet
– LemonSnippet, Commented Jun 5, 2018 at 19:35
Please show some actual (possibly abbreviated) values, so one can test a recovery strategy. — lenz
– lenz, Commented Jun 5, 2018 at 20:58
@lenz Apologies. There was sensitive data and I couldn't put it here. — LemonSnippet
– LemonSnippet, Commented Jun 7, 2018 at 6:13

LemonSnippet · Accepted Answer · 2018-06-07 13:39:10Z

3

Thank you all for your help.

After lots of searching on Stackoverflow coupled with testing on my local machine I was able to drill it down to this.

The object (a['data']) that was being passed in had some values that were not utf-8 decodable.

It was in the form of b'xxxsknoen'

I ended up deleting the b and the quotes in the front and end and then converting it to an str.

var = base64.b64decode(str(a['data'])[2:-1]).decode('utf-8')
a.update(json.loads(var))

edited Jun 7, 2018 at 13:39

answered Jun 7, 2018 at 6:12

LemonSnippet

1151 gold badge3 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Error in json.loads() for data that has base64 decoding applied

the JSON object must be str, not 'bytes'

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

the JSON object must be str, not 'bytes'

1 Answer 1

Comments

Linked

Related