2

I call repetitively - during a minimisation process - a function that requires big arrays. Here is a dummy example

def foo(N,a):
    big_array = np.mgrid[0:N,0:N]
    b = np.fft.fft2(big_array[0]**a) #some heavy computation
    return b

During the minimisation process, the array size N doesn't change, so I would like to use the same array to avoid useless computation and memory allocation.

Also I would like the function foo to be self-consistent, meaning that I don't want another function to create the array and give it to foo during the minimisation process.

Given these requirements, I was thinking to use a callable object with the array as an attribute. What do you think about this? Is there a more pythonic way to do?

Improve this question
2
  • 1
    you mean you want to avoid the allocation or the computation? Commented Nov 29, 2018 at 9:29
  • I want to avoid the useless repetition of the line big_array = np.mgrid[0:N,0:N]. The goal is to make the program faster Commented Nov 29, 2018 at 9:37

1 Answer 1

Reset to default
5

a self contained approach (without global variable) would be to use a mutable default argument (that you shouldn't call your function with) to memoize the previously allocated arrays given their size

if the array size isn't in the dictionary, create it, and add it.

def foo(N,a,dict_container={}):
    if N in dict_container:
        big_array = dict_container[N]
    else:
        big_array = np.mgrid[0:N,0:N]
        dict_container[N] = big_array

    b = np.fft.fft2(big_array[0]**a) #some heavy computation
    return b

The main problem with this approach is that it disables the garbage collector for this array, so if N changes too much, you can have memory exhaustion. Same technique, but using a LRU cache can solve the issue:

from functools import lru_cache
@lru_cache(maxsize=32)  # max 32 elements in cache
def get_matrix(N):
    return np.mgrid[0:N,0:N]

def foo(N,a):
    big_array = get_matrix(N)
    b = np.fft.fft2(big_array[0]**a) #some heavy computation
    return b

(don't define get_matrix inside foo or the cache is going to be reinitialized at each call)

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

What happens if N is not an integer but a numpy array??? Indeed my get_matrix function might take an array as input and return another array... Previous methods both return an unhashable type error
if N is an array, then you want to build an array of zeros given the array dimensions right? in that case, the key would be a tuple of the array dimensions (which is hashable). If you need the contents of the array, then it's going to be difficult...
Yes I need the content of the 2D input array to build a 2D output array :/
Maybe it is possible to build a homemade decorator in the lru_cache fashion. However instead of using a key-value system as in lru_cache, it will parse one by one all the previous inputs. It might work if the number of cached inputs is not too high.
If the same 2D input arrays are being passed to your function (and you don't make copies of those arrays) you could use id as the key of your dictionary (to identify existing objects). This is a trivial change in the first example (and also works with small integers since they're unique from -5 to 256 in CPython)

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.