Getting numpy Memory Error on (relatively) small matrix

Question

I am trying to create a numpy array of certain dimensions however I am getting a Memory Error.

no_of_frames = 1404 
no_of_cells = 136192
original_vals = np.zeros((no_of_frames, no_of_cells), dtype=np.float32)

The error I am getting is:

difference_array = np.zeros((no_of_frames, no_of_cells), dtype=np.float32)
    MemoryError

According to my calculations 1404 x 136192 * 4 is ~729 MB. Which seems pretty reasonable. The machine I am running this code on has 8 GB of RAM. So why am I getting this error?

I would greatly appreciate any help with this. Thanks!

Joran Beasley · Accepted Answer · 2015-09-25 19:43:12Z

1

if you are working in 32bit python you are limited to 32bit address space (~2GB)

if you have other stuff going on it you maybe are exceeding this limit

in addition numpy requires contiguous memory space in order to create its lists ... this means it has to find an uninterupted 768MB block of ram (which is kind of hard)

answered Sep 25, 2015 at 19:43

Joran Beasley

114k13 gold badges167 silver badges187 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user3057470 Over a year ago

Thank you I guess I will have to shift to 64-bit Python.

Dr Xorile · Accepted Answer · 2015-09-25 19:36:58Z

I just tried the code you gave on my 4GB machine with no problem:

Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:43:06) [MSC v.1600 32 bit     (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> no_of_frames = 1404
>>> no_of_cells = 136192
>>> import numpy as np
>>> original_vals = np.zeros((no_of_frames, no_of_cells), dtype=np.float32)
>>> original_vals+=1
>>>

I then performed the calculation (original_vals+=1), which took about 10 seconds, but was also successful. So I don't know if this is a python problem as such. Are you working on win32, win64, linux or some other system?

I am working on a 64-bit Windows system however my mistake is I used 32-bit Python.

Collectives™ on Stack Overflow

Getting numpy Memory Error on (relatively) small matrix

2 Answers 2

1 Comment

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Related