2

I have a 2-d numpy array (MxN) and two more 1-d arrays (Mx1) that represent starting and ending indices for each row of the 2-d array that I'd like to sum over. I'm looking for the most efficient way to do this in a large array (preferably without having to use a loop, which is what I'm currently doing). An example of what i'd like to do is the following.

>>> random.seed(1234)
>>> a = random.rand(4,4)
>>> print a
[[ 0.19151945  0.62210877  0.43772774  0.78535858]
 [ 0.77997581  0.27259261  0.27646426  0.80187218]
 [ 0.95813935  0.87593263  0.35781727  0.50099513]
 [ 0.68346294  0.71270203  0.37025075  0.56119619]]
>>> b = array([1,0,2,1])
>>> c = array([3,2,4,4])
>>> d = empty(4)
>>> for i in xrange(4):
    d[i] = sum(a[i, b[i]:c[i]]) 

>>> print d
[ 1.05983651  1.05256841  0.8588124   1.64414897]

My problem is similar to the following question, however, I don't think the solution presented there would be very efficient. Numpy sum of values in subarrays between pairs of indices In that question, they are wanting to find the sum of multiple subsets for the same row, so cumsum() can be used. However, I will only be finding one sum per row, so I don't think this would be the most efficient means of computing the sum.

Edit: I'm sorry, I made a mistake in my code. The line inside the loop previously read d[i] = sum(a[b[i]:c[i]]). I forgot the index for the first dimension. Each set of starting and ending indices corresponds to a new row in the 2-d array.

Improve this question
2
  • Doing a hefty vectorized operation each iteration shouldn't be that slow. Have you profiled your code and found that the loop overhead is large? Commented Nov 20, 2012 at 15:33
  • The summation is typically over 20-30 elements, but is done 1200 times for each function call (1200x1200 2-d array). Also, the function is called around 600 times per simulation, with quite a few simulations performed, so the looping becomes non-negligible. Commented Nov 20, 2012 at 16:52

1 Answer 1

Reset to default
2

You could do something like this:

from numpy import array, random, zeros
random.seed(1234)
a = random.rand(4,4)
b = array([1,0,2,1])
c = array([3,2,4,4])

lookup = zeros(len(a) + 1, a.dtype)
lookup[1:] = a.sum(1).cumsum()
d = lookup[c] - lookup[b]
print d

This might help if your b/c arrays are large and the windows you're summing over are large. Because your windows might overlap, for example 2:4 and 1:4 are mostly the same, you're essentially repeating operations. By taking the cumsum as a per-processing step you reduce the number of repeated operations and you may save time. This won't help much if your windows are small and b/c are small, mostly because you'll be summing parts of the a matrix that you don't much care about. Hope that helps.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

I'm sorry, this addresses the question when it had a mistake in the code. I've corrected the code and posted the new output, which is what I'm trying to achieve.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.