Parallelize python for-loop

Question

I would like to parallelize the following python loop using a Mac Pro with 2 Quad-Core processors.

result_list = []
for a in a_range:
    for b in b_range:
        for c in c_range:
            result = call_fortran_program(a, b, c)
            result_list.append(result)

In my searches I've run across terms like Cython and GIL but it's still not clear to me on how to proceed.

Answer 1

from itertools import product
from multiprocessing import Pool

with Pool(processes=4) as pool:  # assuming Python 3
    pool.starmap(print, product(range(2), range(3), range(4)))

Answer 2

Try ProcessPoolExecutor

This can circumvent GIL lock by creating multiple processes.

An example from the website:

import concurrent.futures
import math

PRIMES = [
    112272535095293,
    112582705942171,
    112272535095293,
    115280095190773,
    115797848077099,
    1099726899285419]

def is_prime(n):
    if n % 2 == 0:
        return False

    sqrt_n = int(math.floor(math.sqrt(n)))
    for i in range(3, sqrt_n + 1, 2):
        if n % i == 0:
            return False
    return True

def main():
    with concurrent.futures.ProcessPoolExecutor() as executor:
        for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
            print('%d is prime: %s' % (number, prime))

if __name__ == '__main__':
    main()

Answer 3

You can use asyncio. (Documentation can be found [here][1]). It is used as a foundation for multiple Python asynchronous frameworks that provide high-performance network and web-servers, database connection libraries, distributed task queues, etc. Plus it has both high-level and low-level APIs to accomodate any kind of problem.

import asyncio

def background(f):
    def wrapped(*args, **kwargs):
        return asyncio.get_event_loop().run_in_executor(None, f, *args, **kwargs)

    return wrapped

@background
def your_function(argument):
    #code

Now this function will be run in parallel whenever called without putting main program into wait state. You can use it to parallelize for loop as well. When called for a for loop, though loop is sequential but every iteration runs in parallel to the main program as soon as interpreter gets there.

For your case you can do:

import asyncio
import time
from itertools import product


def background(f):
    def wrapped(*args, **kwargs):
        return asyncio.get_event_loop().run_in_executor(None, f, *args, **kwargs)                                   
    return wrapped
    
@background
def call_fortran_program(a, b, c):
    time.sleep(max(0,b-a+c)) # Added Sleep to better demonstrate parallelization
    print(f"function called for {a=}, {b=}, {c=}\n", end='')
    return (a, b, c)

a_range, b_range, c_range = range(1,4), range(1,4), range(1,4)

loop = asyncio.get_event_loop()          # Have a new event loop

looper = asyncio.gather(*[call_fortran_program(a, b, c) for a in a_range for b in b_range for c in c_range])                   # Run the loop
                             
result_list = loop.run_until_complete(looper) # Wait until finish

print(result_list)

This gives following output:

function called for a=1, b=1, c=1
function called for a=1, b=2, c=1
function called for a=1, b=1, c=2
function called for a=2, b=1, c=1
function called for a=1, b=3, c=1
function called for a=1, b=2, c=2
function called for a=1, b=1, c=3
function called for a=2, b=1, c=2
function called for a=1, b=3, c=2
function called for a=1, b=2, c=3
function called for a=2, b=1, c=3
function called for a=2, b=2, c=1
function called for a=3, b=1, c=1
function called for a=3, b=1, c=2
function called for a=3, b=2, c=1
function called for a=2, b=2, c=2
function called for a=2, b=3, c=1
function called for a=3, b=2, c=2
function called for a=3, b=1, c=3
function called for a=2, b=2, c=3
function called for a=1, b=3, c=3
function called for a=3, b=3, c=1
function called for a=2, b=3, c=2
function called for a=3, b=2, c=3
function called for a=3, b=3, c=2
function called for a=2, b=3, c=3
function called for a=3, b=3, c=3
[(1, 1, 1), (1, 1, 2), (1, 1, 3), (1, 2, 1), (1, 2, 2), (1, 2, 3), (1, 3, 1), (1, 3, 2), (1, 3, 3), (2, 1, 1), (2, 1, 2), (2, 1, 3), (2, 2, 1), (2, 2, 2), (2, 2, 3), (2, 3, 1), (2, 3, 2), (2, 3, 3), (3, 1, 1), (3, 1, 2), (3, 1, 3), (3, 2, 1), (3, 2, 2), (3, 2, 3), (3, 3, 1), (3, 3, 2), (3, 3, 3)]