How to save a list of numpy arrays into a single file and load file back to original form [duplicate]

Question

I am currently trying to save a list of numpy arrays into a single file, an example of such a list can be of the form below

import numpy as np
np_list = []
for i in range(10):
    if i % 2 == 0:
        np_list.append(np.random.randn(64))
    else:
        np_list.append(np.random.randn(32, 64))

I can combine all of them using into a single file using savez by iterating through list but is there any other way? I am trying to save weights returned by the function model.get_weights(), which is a list of ndarray and after retrieving the weights from the saved file I intend to load those weights into another model using model.set_weights(np_list). Therefore the format of the list must remain the same. Let me know if anyone has an elegant way of doing this.

the answer here seems to do the trick NumPy save some arrays at once — DVK
– DVK, Commented Feb 13, 2020 at 2:07
Does this answer your question? NumPy save some arrays at once — AMC
– AMC, Commented Feb 13, 2020 at 2:26

marcos · Accepted Answer · 2023-02-09 18:43:45Z

20

I would go with np.save and np.load because it's platform-independent, faster than savetxt and works with lists of arrays, for example:

import numpy as np

a = [
    np.arange(100),
    np.arange(200)
]
np.save('a.npy', np.array(a, dtype=object), allow_pickle=True)
b = np.load('a.npy', allow_pickle=True)

This is the documentation for np.save and np.load. And in this answer you can find a better discussion How to save and load numpy.array() data properly?

Edit

Like @AlexP mentioned numpy >= v1.24.2 does not support arrays with different sizes and types, so that's why the casting is necessary.

edited Feb 9, 2023 at 18:43

answered Feb 13, 2020 at 2:06

marcos

4,5101 gold badge13 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

DVK Over a year ago

This does not work for a list of numpy arrays

hpaulj Over a year ago

Careful with this approach. save will do np.array(a) before saving. So b will be an array, not a list. And as others have found out, making an array from a list of arrays has its hidden dangers. Don't do it naively.

marcos Over a year ago

@hpaulj oh, that's True! Thanks for the insight.

J Agustin Barrachina Over a year ago

It does not let me answer, so I go here. It did not worked for me. The solution was to use np.savez. I would do the complete answer if I could comment.

AlexP Over a year ago

using numpy 1.24.2 and it no longer works if storing arrays of different size or different type. Use instead: np.save('a.npy', np.array(a,dtype=object), allow_pickle=True)

|

Collectives™ on Stack Overflow

How to save a list of numpy arrays into a single file and load file back to original form [duplicate]

1 Answer 1

Edit

7 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Edit

7 Comments

Linked

Related