Python3: Parallel Processing using Yield generator

Question

I will have one function that will yield values and another that will perform some operation on those yielded values. I would like to do the operation using parallel processing (i.e. concurrent.futures.ProcessPoolExecutor or Multiprocessing). The order of the process is important i.e. first output should be of the first input. Pseudocode is as below:

def square(x):
    return x**2

def numbers():
    for i in range(1,10):
        yield i

if __name__ == '__main__':
    with concurrent.futures.ProcessPoolExecutor(4) as executor:        
       for i in executor.map(square, numbers):
           print(i)

I have following questions that I didn't find much explanation for:

1) How to map yield generator and the function

2) If ProcessPoolExecutor a right choice since it asynchronous and would mess up the order of input of output.

3) How to use Multiprocessing Pool and generator. Pool map method is not printing any output for me. I am having a tough time to understanding the application of Multiprocessing Pool.

if __name__ == '__main__':
    with mp.Pool(4) as p:        
        print(p.map(square, numbers))

I would be grateful if someone can help me understand parallel processing.

P.S. I know that if I convert the generator function into a list of numbers and map the function square, it is easy to use and understand multiprocessing; and also everything works fine but I don't want to load entire data in the memory.

mehdix · Accepted Answer · 2018-03-07 15:35:22Z

1

You have an error in both your examples. You have to call the numbers function to actually generate a generator:

if __name__ == '__main__':
    with concurrent.futures.ProcessPoolExecutor(4) as executor:        
       for i in executor.map(square, numbers()):
           print(i)

and

if __name__ == '__main__':
    with mp.Pool(4) as p:        
        print(p.map(square, numbers()))

answered Mar 7, 2018 at 15:35

mehdix

5,2042 gold badges31 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Krishnang K Dalal Over a year ago

Oh! That was simple and this solved a very big problem for me. Thank you. I actually didn't know how to exactly go about it. I was under the impression that I have to write a loop and call the numbers in that loop to get the values. But this is neat.

mehdix Over a year ago

Good luck with the rest of your concurrent journey :)

Collectives™ on Stack Overflow

Python3: Parallel Processing using Yield generator

1 Answer 1

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Related