Is there a better way to use cython when looking to speed up Python?

Question

I have a large numpy array with the following structure:

array([['A', 0.0, 0.0],
       ['B2', 1.0, 0.0],
       ['B4', 2.0, 3.0],
       ['AX1', 3.0, 1.0],
       ['C2', 0.0, 2.0],
       ['D3', 2.0, 1.0],
       ['X4', 3.0, 8.0],
       ['BN', 2.0, 9.0],
       ['VF', 12.0, 25.0],
       ['L', 1.0, 3.0],
       ...,
       ['s', 2.0, 27.0],
       ['P', 0.0, 0.0]], dtype=object)

I'm using cython to try and speed up the processing as much as possible. The argument dataset in the code below is the above array.

%%cython

cpdef permut1(dataset):
  cdef int i
  cdef int j
  cdef str x
  cdef str v

  xlist = []
  for i, x in enumerate(dataset[:,0]):
    for j, v in enumerate(dataset[:,0]):
      xlist.append((x,v,dataset[i][1], dataset[j][2]))
  return xlist

However, when running the above code with and without cython I get the following times:

without cython: 0:00:00.945872

with cython: 0:00:00.561925

Any ideas how I can use cython to speed this up even more?

thanks

cdef str x and v is probably making it (slightly) worse. You don't use the type information at all so all you're adding is a pointless type check. That may be true for i and j too; I'm not sure. dataset[i][1] -> dataset[i,1] (it may be maginally quicker since it's one operation). — DavidW
– DavidW, Commented Mar 19, 2021 at 15:10
Object dtype arrays are just glorified (or debased?) lists. You get best cython speed improvements when using arrays, or code that can be readily translated into c. Lists and arrays that have to reference other Python objects require a lot of Python code,even in cython. There is a way of getting an annotated html display of your cython code. Bright yellow in that display indicates Python heavy lines. — hpaulj
– hpaulj, Commented Mar 19, 2021 at 16:37

cidermole · Accepted Answer · 2021-03-19 15:43:49Z

Generally with numpy, you want to:

Put the same data type into an array (avoid dtype=object and use a separate array for the strings). Otherwise every element access must internally test for the data type, and that will slow things down. This is equally true for cython.
Avoid element-wise access and use only operations on entire arrays instead. For your case, consider building up the indices in an integer array and express the indexing of your input array as one operation.

E.g.:

a = np.array(..., dtype=np.float) # input number columns only from above
fa = a.flatten()  # helps to use 1d indices
far = fa.reshape((fa.shape[0], 1))  # make 2d for hstack()
idxs = np.indices((a.shape[0], a.shape[0]))
idxs1 = idxs[0].flatten()  # 0,0,0,1,1,1,...
idxs2 = idxs[1].flatten()  # 0,1,0,1,...
np.hstack((far[idxs1], far[idx2]))

No cython needed (unless you really need complex element-wise calculation).

Since you previously iterated with O(n^2) operations, the above should also work out to a speedup even if you first have to convert your input array to this format.

Collectives™ on Stack Overflow

Is there a better way to use cython when looking to speed up Python?

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related