PERF: uses bincount instead of hash table in categorical value counts

[behzadnouri]

closes #10804

In [1]: np.random.seed(2718281)

In [2]: n = 500000

In [3]: u = int(0.1*n)

In [4]: arr = ["s%04d" % i for i in np.random.randint(0, u, size=n)]

In [5]: ts = pd.Series(arr).astype('category')

In [6]: %timeit ts.value_counts()
10 loops, best of 3: 82.7 ms per loop

on branch:

In [6]: %timeit ts.value_counts()
10 loops, best of 3: 31.3 ms per loop

[jreback]

I think the soln in the issues is faster than this no?

[behzadnouri]

I get

In [9]: %timeit Series(np.arange(len(ts.cat.categories)),ts.cat.categories).map(ts.cat.codes.value_counts()).order(ascending=False)
10 loops, best of 3: 28.3 ms per loop

but this does not check for nulls, and the index is not categorical.

[behzadnouri]

with dropna=False the branch performs better:

In [9]: %timeit ts.value_counts(dropna=True)
10 loops, best of 3: 32.3 ms per loop

In [10]: %timeit ts.value_counts(dropna=False)
10 loops, best of 3: 25.7 ms per loop

In [11]: %timeit Series(np.arange(len(ts.cat.categories)),ts.cat.categories).map(ts.cat.codes.value_counts()).order(ascending=False) 
10 loops, best of 3: 29.5 ms per loop

[jreback]

wow, this does even better!

In [7]: %timeit ts.value_counts(dropna=True)
100 loops, best of 3: 11 ms per loop

In [8]: %timeit ts.value_counts(dropna=False)
100 loops, best of 3: 9.53 ms per loop

In [9]: %timeit Series(np.arange(len(ts.cat.categories)),ts.cat.categories).map(ts.cat.codes.value_counts()).order(ascending=False) 
100 loops, best of 3: 17.4 ms per loop

[jreback]

ping when green

[jorisvandenbossche]

Maybe worth adding a benchmark?

[behzadnouri]

I will add benchmark later today

[behzadnouri]

added the benchmark, all green.

[jreback]

These should have only 1 action per timing function (so make 2 functions)

[behzadnouri]

why should it be only 1 action?

[jorisvandenbossche]

You get a timing per function. So if you want to track performance of both with dropna True and False, it has to be in two functions.

[behzadnouri]

added separate calls

[jreback]

thank you sir!