Most efficient property to hash for numpy array

You can try xxhash via its Python binding. For large arrays this is much faster than hash(x.tostring()).

Example IPython session:

>>> import xxhash
>>> import numpy
>>> x = numpy.random.rand(1024 * 1024 * 16)
>>> h = xxhash.xxh64()
>>> %timeit hash(x.tostring())
1 loops, best of 3: 208 ms per loop
>>> %timeit h.update(x); h.intdigest(); h.reset()
100 loops, best of 3: 10.2 ms per loop

And by the way, on various blogs and answers posted to Stack Overflow, you'll see people using sha1 or md5 as hash functions. For performance reasons this is usually not acceptable, as those "secure" hash functions are rather slow. They're useful only if hash collision is one of the top concerns.

Nevertheless, hash collisions happen all the time. And if all you need is implementing __hash__ for data-array objects so that they can be used as keys in Python dictionaries or sets, I think it's better to concentrate on the speed of __hash__ itself and let Python handle the hash collision[1].

[1] You may need to override __eq__ too, to help Python manage hash collision. You would want __eq__ to return a boolean, rather than an array of booleans as is done by numpy.

If your np.array() is small and in a tight loop, then one option is to skip hash() completely and just use np.array().data.tobytes() directly as your dict key:

grid  = np.array([[True, False, True],[False, False, True]])
hash  = grid.data.tobytes()
cache = cache or {}
if hash not in cache:
    cache[hash] = function(grid)
return cache[hash]

Coming late to the party, but for large arrays, I think a decent way to do it is to randomly subsample the matrix and hash that sample:

def subsample_hash(a):
    rng = np.random.RandomState(89)
    inds = rng.randint(low=0, high=a.size, size=1000)
    b = a.flat[inds]
    b.flags.writeable = False
    return hash(b.data)

I think this is better than doing hash(str(a)), because the latter could confuse arrays that have unique data in the middle but zeros around the edges.

What kind of data do you have?

• array-size
• do you have an index several times in the array

If your array only consists of permutation of indices you can use a base-convertion

(1, 0, 2) -> 1 * 3**0 + 0 * 3**1 + 2 * 3**2 = 10(base3)

and use '10' as hash_key via

import numpy as num

base_size = 3
base = base_size ** num.arange(base_size)
max_base = (base * num.arange(base_size)).sum()

hashed_array = (base * array).sum()

Now you can use an array (shape=(base_size, )) instead of a dict in order to access the values.

While the other answers are good, intelligent analysis is almost always the way to go with this sort of challenge. When we know what it is that makes an array uniquely interesting, then it can often be encoded in a meaningful way. There are major advantages-

• We can determine when a clash is a clash - for example, it might be that 1e-20 is close enough to 2e-20 that we want them to clash.
• We can choose an encoding which compactly represents the array, and is thereby not costly.
• We aren't using a sledgehammer to crack a nut.

We must have an idea about what is being captured - and so domain expertise is required, whereas general purpose hashing obviates such a need.

For example, we might be attempting to manage a large array of polygons (or polyhedra) in space. If we know that each polygon must have a distinct centroid, we can 'hash' merely by using that centroid.

import numpy as np

np.random.seed(42)
count = 100000
polys = np.random.rand(count, 4, 2) # 100k 2d polygons.
keys = np.mean(polys, axis=1).round(4)  # four decimal places...
hashes = set()
unique = []
for k, p in zip(keys, polys):
    if tuple(k) not in hashes:
        hashes.add(tuple(k))
        unique.append(p)
print(f'Unique values: {len(unique)}, {count-len(unique)} culled')
#  Unique values: 99808, 192 culled