Parallel processing in python for calculation of a function

Question

I have a heavy code and running it takes a long time. I ran into the following code, but I really don't know how it does work. Consider def sigma(b) is a huge function in the middle of a code script. This part is the main reason of slowing the code running. We use the results of sigma function in the other parts of code. I just put it in the following code.

from multiprocessing import Pool
from multiprocessing import Process
import multiprocessing
def sigma(b):
    n=0
    for i in range (1,550):
        n=n+i+b
        return n
p = multiprocessing.Process(target=sigma)
p.start()

print(sigma(2))

Can any one help me please?

Answer 1

One way to do it (after fixing the indentation of sigma):

# -*- coding: utf-8 -*-
import multiprocessing as mp

def sigma(b):
    n=0
    for i in range (1,550):
        n=n+i+b
    return n

if __name__ == '__main__':
    inputs_b = [1, 2, 3, 4]

    with mp.Pool(processes = 2) as p:
        res = p.map(sigma, inputs_b)

The only issue with multiprocessing is that you can't run it in an IDE (like spyder), thus you need to save the results and retrieve it later.

It can be done with numpy, pandas, pickle, or others.

Then you might need to have multiple arguments. In this case, use starmap():

# -*- coding: utf-8 -*-
import multiprocessing as mp

def sigma(a, b):
    n=0
    for i in range (1,550):
        n=n+i+b+a
    return n

if __name__ == '__main__':
    inputs_b = [(a,b) for a in range(5) for b in range(6, 10)]

    with mp.Pool(processes = 2) as p:
        res = p.starmap(sigma, inputs_b)

N.B: processes = N gives the number of processes to open. It is recommended to use the number of physical CPUs or the number of CPUs-1.

EDIT2: Your dummy example is a very simple case. You have 2 options: write your function to do an elementary task and parallelize the elementary tasks OR take your big function running for 72 hours and run 4 or more at the same time on different input.

You also need to make sure that the processes do not use shared resources or you'll need to use more complex implementation.

Finally, using multiprocessing on functions which generates a lot of data might end in a Memory error (Not enough RAM). This will depend on the application.