Numpy random memory error

Question

Hi I have a simple line that creates a random array for a rather large dataset:

import numpy as np
import random
N=276233
L=138116

np.random.random([L,N])

But i get this error:

Traceback (most recent call last):
  File "<string>", line 3 (23), in <module>
  File "mtrand.pyx", line 760, in mtrand.RandomState.random_sample (numpy\random\mtrand\mtrand.c:5713)
  File "mtrand.pyx", line 137, in mtrand.cont0_array (numpy\random\mtrand\mtrand.c:1300)
MemoryError

What is the solution and what is the limit of the array ?

If you can use a smaller integer type rather than doubles you could reduce the memory foot print by quite a bit. However, depending on the goals of your analysis / data this may not be possible. — GWW
– GWW, Commented Jan 19, 2015 at 20:40

NPE · Accepted Answer · 2015-01-19 20:47:54Z

9

You are trying to create an array that would require 284GB of memory:

In [16]: L * N * 8 / (1024. ** 3)
Out[16]: 284.25601890683174

Either buy a lot more RAM (and make sure your system can handle it) or find a way to not have to generate a 276,233x138,116 matrix.

edited Jan 19, 2015 at 20:47

answered Jan 19, 2015 at 20:37

NPE

503k114 gold badges970 silver badges1k bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

chad Over a year ago

Hmm...How did you get the number 284GB ? I have N*L=3.81x10^10 bits / 8 bits/Byte = 4.7GB ? Am i wrong?

GWW Over a year ago

276233 * 138116 * 8 / (1024 ^ 3). Each double uses 8 bytes.

chad Over a year ago

Ah okay thanks! I thought each they're a bit each. :( I guess I can't use this method for large dataset which requires a random matrix ...

NPE Over a year ago

@Arbitel: Are you sure you need to generate the entire matrix at once?

chad Over a year ago

It goes back to this question I posted earlier : stackoverflow.com/questions/28015281/numpy-optimization I basically had to compare the neighbouring elements. After that I would have to do a row-wise summation. I guess I have to?

Collectives™ on Stack Overflow

Numpy random memory error

1 Answer 1

5 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Linked

Related