Array for Large Data

Question

I need to form a 2D matrix with total size 2,886 X 2,003,817. I try to use numpy.zeros to make a 2D zero element matrix and then calculate and assign each element of Matrix (most of them are zero son I need to replace few of them).

but when I try numpy.zero to initialize my matrix I get the following memory error:

 C=numpy.zeros((2886,2003817)) "MemoryError"

I also try to form the matrix without initialization. Basically I calculate the element of each row in each iteration of my algorithm and then

C=numpy.concatenate((C,[A]),axis=0)

in which C is my final matrix and A is the calculated row at the current iteration. But I find out this method takes a lots of time, I am guessing it is because of using numpy.concatenate(?)

could you please let me know if there is a way to avoid memory error in initializing my matrix or is there any better method or suggestion to form the matrix in this size?

Thanks, Amir

The default datatype for numpy.zeros is numpy.float64 (an 8-byte float). Therefore the base memory requirement for your matrix is 2,886 * 2,003,817 * 8 bytes == 44.1 GB of RAM. — Hugh Bothwell
– Hugh Bothwell, Commented Mar 31, 2014 at 2:17

Akavall · Accepted Answer · 2014-03-31 02:15:55Z

2

If your data has a lot of zeros in it, you should use scipy.sparse matrix.

It is a special data structure designed to save memory for matrices that have a lot of zeros. However, if your matrix is not that sparse, sparse matrices start to take up more memory. There are many kinds of sparse matrices, and each of them is efficient at one thing, while inefficient at another thing, so be careful with what you choose.

answered Mar 31, 2014 at 2:15

Akavall

86.8k58 gold badges214 silver badges260 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Array for Large Data

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related