Memmap in MATLAB for huge arrays

Question

I want to create a memmap in MATLAB. In python I could do this by:

ut = np.memmap('my_array.mmap', dtype=np.float64, mode='w+', shape=(140000,3504))

Then I use it as a normal array, the OS ensured my memory never overflowed. How to do this in MATLAB?

From the docs it seems it wants me to create some array in MATLAB first then write it to a file and read using memmap!

Matlab docs are not clear enough: Please provide an example of creating an random array of size (140000,15000) and multiply it some other similar matrix.

What I understand from the documentation is that you cannot create one array which is bigger than your RAM, e.g. one 200GB array, but you can create e.g. 200 1GB arrays, which you store in a temporary location on disk which is accessible in a slower way than RAM, but faster than a full write-to-file script. — Adriaan
– Adriaan, Commented Oct 21, 2015 at 11:05
@IKavanagh Matlab docs seem inefficient in explaining it in detail. Adriaan comment is very helpful for start. Still have doubts on to apply a elementary-wise operation. — Abhishek Bhatia
– Abhishek Bhatia, Commented Oct 21, 2015 at 11:33

Daniel · Accepted Answer · 2015-10-21 12:10:49Z

6

+25

You have to create an empty file first, then use memmapfile:

size=[140000,3504];
filesize=0;
datatype='float64';
filename='my_array.dat';
fid=fopen(filename,'w+');
max_chunk_size=1000000;
%fills an empty file
while filesize<prod(size)
    to_write=min(prod(size)-filesize,max_chunk_size);
    filesize=filesize+fwrite(f, zeros(to_write,1), datatype);
end   
fclose(fid);
m = memmapfile(filename,'Format','double', 'Writable',true);

answered Oct 21, 2015 at 12:10

Daniel

36.7k3 gold badges38 silver badges73 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Abhishek Bhatia Over a year ago

Thanks for the reply!! Can you please also tell how to perform some operation on the array.

Daniel Over a year ago

What kind of operations?

Abhishek Bhatia Over a year ago

Matlab says the memmap canbe used just like other normal arrays. So how to appl any operation say transpose on m.Data without blowing my RAM.

Daniel Over a year ago

Transposing requires to reorganize the full data, that's something you typically want to avoid at all costs when dealing with such data sizes. Try to apply MapReduce or similar to create multiple steps dealing with smaller data.

Ander Biguri · Accepted Answer · 2015-10-21 10:14:15Z

2

I think what you are looking for is the function memmapfile

Example:

m = memmapfile('my_array.dat','Format','double', 'Writable',true)

edited Oct 21, 2015 at 10:14

Ander Biguri

35.7k12 gold badges77 silver badges125 bronze badges

answered Oct 19, 2015 at 3:01

gregswiss

1,4569 silver badges20 bronze badges

5 Comments

Abhishek Bhatia Over a year ago

Can you provide an example of creating random array of size '5000x153600'.

gregswiss Over a year ago

that function won't create a file by itself, aifk you need to first create a file of sufficient size. For example with a combination of fopen and fprintf. You can do so in a loop if the array is too big to hold in memory.

Abhishek Bhatia Over a year ago

I tried this :` >> m = memmapfile('my_array.dat','Format','double', 'Writable',true); Error using memmapfile/hChangeFilename (line 351) Cannot access file "my_array.dat": No such file or directory. Error in memmapfile (line 568) obj = hChangeFilename(obj, filename);`. Can you please explain by example.

Daniel Over a year ago

You have to create the file first.

Abhishek Bhatia Over a year ago

@Daniel that will crash my RAM.

Collectives™ on Stack Overflow

Memmap in MATLAB for huge arrays

2 Answers 2

4 Comments

5 Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

5 Comments

Related