38

I have a huge numpy 3D tensor which is stored in a file on my disk (which I normally read using np.load). This is a binary .npy file. On using np.load, I quickly end up using most of my memory.

Luckily, at every run of the program, I only require a certain slice of the huge tensor. The slice is of a fixed size and its dimensions are provided from an external module.

What's the best way to do this? The only way I could figure out is somehow storing this numpy matrix into a MySQL database. But I'm sure there are much better / easier ways. I'll also be happy to build my 3D tensor file differently if it will help.

Does the answer change if my tensor is sparse in nature?

Improve this question
5
  • File type'd help. Commented Mar 10, 2017 at 20:46
  • 1
    It's a binary file, .npy. Saved using np.save Commented Mar 10, 2017 at 20:49
  • 1
    Good question. I don't know of any tool for this (but there may well be one). Is the slice always along the same axis? Commented Mar 10, 2017 at 20:55
  • Here's a place to start. what are the dimensions / dtype of the tensor? Commented Mar 10, 2017 at 20:58
  • chunky3d works well for sparse 3D data. Documentation is scarce but there are some neat functionalities in it. Commented May 13, 2024 at 15:30

1 Answer 1

Reset to default
53

use numpy.load as normal, but be sure to specify the mmap_mode keyword so that the array is kept on disk, and only necessary bits are loaded into memory upon access.

mmap_mode : {None, ‘r+’, ‘r’, ‘w+’, ‘c’}, optional If not None, then memory-map the file, using the given mode (see numpy.memmap for a detailed description of the modes). A memory-mapped array is kept on disk. However, it can be accessed and sliced like any ndarray. Memory mapping is especially useful for accessing small fragments of large files without reading the entire file into memory.

The modes are described in numpy.memmap:

mode : {‘r+’, ‘r’, ‘w+’, ‘c’}, optional The file is opened in this mode: ‘r’ Open existing file for reading only. ‘r+’ Open existing file for reading and writing. ‘w+’ Create or overwrite existing file for reading and writing. ‘c’ Copy-on-write: assignments affect data in memory, but changes are not saved to disk. The file on disk is read-only.

*be sure to not use 'w+' mode, as it will erase your file's contents.

Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

Amazing! I don't even know that. This is such an impressive note about numpy, providing SSD is so popular today. :)
Not unfortunately that if you need to read the entire file, just not load it all at once, mmap is not of much help. For example if you create a generator that yields chunks of the data, with the hope that your program never consumes more memory than the cost of a chunk. With mmap, memory used grows and grows as you request more and more chunks to be loaded, without 'releasing' older chunks that you might be done with.
@ely true, however using a generator is a little out of kind for numpy anyway as the preferred method is to take advantage of vectorization rather than iteration. In that instance I would likely use struct to pack the data into a binary file, and numba to jit compile a fast function to read and analyze the data.
@ely It would be great if you could specify a cache size for what to keep in memory with mmap before flushing to disk. (anyone wanna write a pull request??)
For example, when pre-processing very large data sets to feed in as the input to training a neural network. You might not be able to load the whole thing in memory all at once, but you will have to pass every part of the contents through memory at some point, and you might need to perform linear algebra, data cleaning, etc., in vectorized fashion, even for sub-portions of the data which can fit into memory.
|

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.