Load multiple memory mapped files with numpy

Question

I'm trying to load two memory mapped files,

temp = numpy.load(currentDirectory + "\\tmp\\temperature.npy", mmap_mode='r')
salinity = numpy.load(currentDirectory + "\\tmp\\salinity.npy", mmap_mode='r')

but Python throws the following error:

IOError: Failed to interpret file 'C:\\my\\file\\path\\..\\tmp\\salinity.npy' as a pickle

When I load either by itself, it works just fine.

The files that are quite large (~500MB), but otherwise I don't think they are notable.

What might the problem be here?

Does the same thing happen with smaller files? It might help if you show the shape and dtype of these files (when loaded individually). — hpaulj
– hpaulj, Commented Mar 8, 2016 at 2:31
Hi @hpaulj , shape = (1, 102, 720, 1440) and type = float32 . I'll give some smaller files a try and see what happens... — rowanwins
– rowanwins, Commented Mar 8, 2016 at 2:46
Hi @hpaulj it appears to work fine when I have two small files... I wonder if Im running out of memory to interpret the file? — rowanwins
– rowanwins, Commented Mar 8, 2016 at 2:53

Mohbat Tharani · Accepted Answer · 2018-03-08 08:48:00Z

1

This works for me. Both files are large than 5GB.

X = np.load(os.path.join(path, '_file1.npy'), mmap_mode='r')
Y = np.load(os.path.join(path, '_file2.npy'), mmap_mode='r')

Which operating system are you using? The problem is not with size of the "npy" files but problem is with "\" in path. change your path as:

path = '/media/gtx1060/DATA/Datasets'

answered Mar 8, 2018 at 8:48

Mohbat Tharani

5701 gold badge9 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Load multiple memory mapped files with numpy

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related