I'm trying to run following code with variable limit in this case being in the range of milions:
limit = 6000000
for i in range(limit):
s[i] = some_function(i)
With some_function being a relative simple function without for loops in it (ex.: 2**i).
How can I make this run faster? (maybe by using numpy?)
This solution of using map() is very fast as compared to for loop
limit = 6000000
def some_function(i):
return i**2
s=list(map(some_function,range(limit)))
map speeds things up that much as wellIf the loop is large and you have other unused processor, please check multiprocessing library as this is python's hacky way to allow parallelization of processes.
The first example in the link is all you need:
from multiprocessing import Pool
n_pools = 5 # You will need to play with this number. Increase or decrease and check performance.
with Pool(n_pools ) as p:
print(p.map(some_function, range(limit)))
If some_function is just 2**i then numpy could speed thing up for you. You'd just call numpy.power:
import numpy as np
elements = np.power(2, range(10000))
As suggested by Benoit Descamps before me, you could speed things up even further when using multiprocessing.
from multiprocessing import Pool
def f(x):
return x**2
p = Pool(5)
p.map(f, range(10000))
multiprocessing VS numpy VS map VS for-loop
$ echo -e "from multiprocessing import Pool\ndef f(x):\n return x**2\np = Pool(5)\np.map(f, range(10000))" | python -m timeit -n 1000
1000 loops, best of 5: 4.83 nsec per loop
$ python -m timeit -n 1000 'import numpy as np; np.power(2, range(10000))'
1000 loops, best of 5: 1.09 msec per loop
$ python -m timeit -n 1000 'list(map(lambda v: v**2, range(10000)))'
1000 loops, best of 5: 2.07 msec per loop
$ python -m timeit -n 1000 'import numpy as np; [np.power(2, i) for i in range(10000)]'
1000 loops, best of 5: 8.92 msec per loop
Lets try make some speed improvment for your example...
import nupmy as np
limit = 6000000
s= np.arange(limit)
s = s**2
print(s[:10])
This using numpy vectorize should make big deal when it comes to python loops speed, it is under the using the Cython module that convert parts of the code to c (or c++ I dont remember...)
some_function6 million times there is no faster way.