Load numpy arrays from compressed bytes

Question

I'm trying to save (using savez_compressed) a bunch of numpy arrays into a BytesIO object and then load them back to a variable. So far, I can save the array's using following code

# Arrays
a = numpy.random.uniform(size=(10,10)) # dtype is float64
b = numpy.random.uniform(size=(10,10)) # dtype is float64

# Create bytes object
buf = io.BytesIO()

# Save the arrays into the buffer
numpy.savez_compressed(buf, a=a, b=b)

# Closed the buffer
buf.close()

I've been trying different methods to load them back. For example

ab = numpy.frombuffer(buf.read(), dtype='float64')

which raises ValueError: buffer size must be a multiple of element size. And trying to load as one would if it was a file

ab = numpy.load(buf)

raises ValueError: Cannot load file containing pickled data when allow_pickle=False and when buf.read() i get ValueError: embedded null byte.

Btw. To close the buffer, you sould use buf.close(), and not buf.seek() — Vishal
– Vishal, Commented Nov 21, 2019 at 10:05

0Valt · Accepted Answer · 2021-04-17 04:18:02Z

1

The following works for me [Numpy version: numpy==1.16.1]:

import numpy
import io

# Arrays
a = numpy.random.uniform(size=(10,10)) # dtype is float64
b = numpy.random.uniform(size=(10,10)) # dtype is float64

# Create bytes object
buf = io.BytesIO()

# Save the arrays into the buffer
numpy.savez_compressed(buf, a=a, b=b)
buf.seek(0)

ab = numpy.load(buf)
print(ab['a'])
print(ab['b'])

# Closed the buffer
buf.close()

edited Apr 17, 2021 at 4:18

0Valt

10.5k9 gold badges41 silver badges65 bronze badges

answered Nov 21, 2019 at 10:09

Vishal

3,3263 gold badges39 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Rikard Olsson Over a year ago

It works!! Although, it doesn't work with my original arrays but they are of dtype "O". I guess that's the problem...

Rikard Olsson Over a year ago

It was the dtype. Thanks!!

Collectives™ on Stack Overflow

Load numpy arrays from compressed bytes

1 Answer 1

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Related