Memory error with numpy. array

Question

I get a memory error when using numpy.arange with large numbers. My code is as follows:

import numpy as np

list = np.arange(0, 10**15, 10**3)
profit_list = []

for diff in list:
    x = do_some_calculation
    profit_list.append(x)

What can be a replacement so I can avoid getting the memory error?

fuglede · Accepted Answer · 2017-06-05 15:01:42Z

1

If you replace list¹ with a generator, that is, you do

for diff in range(10**15, 10**3):
    x = do_some_calculation
    profit_list.append(x)

then that will no longer cause MemoryErrors as you no longer initiate the full list. In this world, though, profit_list will probably by causing issues instead, as you are trying to add 10^12 items to that. Again, you can probably get around that by not storing the values explicitly, but rather yield them as you need them, using generators.

¹: Side note: Don't use list as a variable name as it shadows a built-in.

edited Jun 5, 2017 at 15:01

answered Jun 5, 2017 at 14:45

fuglede

18.3k3 gold badges62 silver badges107 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

user123 Over a year ago

I was also trying to plot(difficulty_list,profit_list). Would this be done the same way?

fuglede Over a year ago

I don't believe there is any way out of that, as matplotlib will act on numpy arrays; why would you ever want to plot 10^12 points in the first place?

DSM Over a year ago

You should think about the size of the numbers you're working with, user123. Setting aside the memory problems, for me, it takes about 20 ms to make an ndarray with 10^7 elements. Let's assume that scales linearly (in the real world it'll slow down even more). That means it'll take more than half an hour just to build it. And that's happening at C speed. It takes me about 10 ns just to execute pass, which means that whatever your loop does, it'll take orders of magnitude longer than 10^12 * 10 * 10^-9ns ~ 3 hours. At best you should work with millions of numbers, not trillions.

Collectives™ on Stack Overflow

Memory error with numpy. array

1 Answer 1

3 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Related