numpy save/load corrupting an array

Question

I am trying to save a large numpy array and reload it. Using numpy.save and numpy.load, the array values are corrupted/change. The shape and data type of the array pre-saving, and post-loading, are the same, but the post-loading array has the vast majority of the values zeroed. The array is (22915,22915), values are float64's, takes 3.94 gb's as a .npy file, and the data entries average about .1 (not tiny floats that might reasonably get converted to zeroes). I am using numpy 1.5.1.

Any help on why this corruption is occurring would be greatly appreciated because I am at a loss. Below is some code providing evidence of the claims above.

In [7]: m
Out[7]: 
      array([[ 0.     ,  0.02023,  0.00703, ...,  0.02362,  0.02939,  0.03656],
             [ 0.02023,  0.     ,  0.0135 , ...,  0.04357,  0.04934,  0.05651],
             [ 0.00703,  0.0135 ,  0.     , ...,  0.03037,  0.03614,  0.04331],
             ..., 
             [ 0.02362,  0.04357,  0.03037, ...,  0.     ,  0.01797,  0.02514],
             [ 0.02939,  0.04934,  0.03614, ...,  0.01797,  0.     ,  0.01919],
             [ 0.03656,  0.05651,  0.04331, ...,  0.02514,  0.01919,  0.     ]])
In [8]: m.shape
Out[8]: (22195, 22195)

In [12]: save('/Users/will/Desktop/m.npy',m)

In [14]: lm = load('/Users/will/Desktop/m.npy')

In [15]: lm
Out[15]: 
       array([[ 0.     ,  0.02023,  0.00703, ...,  0.     ,  0.     ,  0.     ],
              [ 0.     ,  0.     ,  0.     , ...,  0.     ,  0.     ,  0.     ],
              [ 0.     ,  0.     ,  0.     , ...,  0.     ,  0.     ,  0.     ],
              ..., 
              [ 0.     ,  0.     ,  0.     , ...,  0.     ,  0.     ,  0.     ],
              [ 0.     ,  0.     ,  0.     , ...,  0.     ,  0.     ,  0.     ],
              [ 0.     ,  0.     ,  0.     , ...,  0.     ,  0.     ,  0.     ]])
In [17]: type(lm[0][0])
Out[17]: numpy.float64

In [18]: type(m[0][0])
Out[18]: numpy.float64

In [19]: lm.shape
Out[19]: (22195, 22195)

I can't -- 1.5.1 is a dependency for the other libraries I am using. If its fixed in an updated version (which there was some evidence of in the searches I conducted) then perhaps I can try to upgrade, though obviously this might cause more/other problems. — wdwvt1
– wdwvt1, Commented Oct 3, 2012 at 17:21
Couple things to try: 1) look for the position where the array goes to zero, 2) try printing out the last row/column of the array in isolation (to see if it might be a weird printing issue), 3) try using memory mapping by specifying 'r' as a second parameter to load. — nneonneo
– nneonneo, Commented Oct 3, 2012 at 17:25
it wasn't a printing issue, but the 'r' flag works. i had tried some of the other map_mode options but they were not resolving the problem so i abandoned that approach. thanks very much for the help! as far as why load without map_mode wouldnt work, do you think there is a maximum constraint on arrays loaded directly into python? — wdwvt1
– wdwvt1, Commented Oct 3, 2012 at 18:29
@wdwvt1 have you tested to see if those values are actually zero? I ask because I came across a bug in Pandas, which displayed floats less than 0 as 0. even though they were correctly represented in memory. — Louis Thibault
– Louis Thibault, Commented Oct 3, 2012 at 19:45

U2EF1 · Accepted Answer · 2012-10-13 19:02:54Z

1

This is a known issue (note that that links against numpy 1.4). If you really can't upgrade, my advice would be to try to save in a different way (savez, savetxt). If getbuffer is available you can try to write the bytes directly. If all else fails (and you can't upgrade), you can write your own save function pretty easily.

answered Oct 13, 2012 at 19:02

U2EF1

13.3k3 gold badges37 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

numpy save/load corrupting an array

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related