Skip to main content
added 57 characters in body
Source Link
Miguel Alorda
  • 1.4k
  • 7
  • 14

If you are not going to write anything to X (which it seems you are not doing), and just going to read from it, you should be good to just have all processes access the same variable without some locking mechanism.

Now, global is not necesarily the way to go here. You could just declare X on a function scope and pass it to your function using a lambdaHere is different approach:

from itertools import combinations
import numpy as np
from scipy.stats import pearsonr
from multiprocessing import Pool


class MyCalculator:

    def function__init__(self, X):
        self.X = X

    def function(self, cols):
        result = self.X[:, cols]
        x,y = result[:,0], result[:,1]
        result = pearsonr(x,y)
        return result


def main():
    X = np.random.random(100000*10).reshape((100000, 10))
    myCalculator = MyCalculator(X)
    with Pool(processes=4) as P:
        print(P.map(lambda cols: myCalculator.function(X, cols), 
                    combinations(range(X.shape[1]), 2)))


if __name__ == '__main__':
    main()

If you are not going to write anything to X (which it seems you are not doing), and just going to read from it, you should be good to just have all processes access the same variable without some locking mechanism.

Now, global is not necesarily the way to go here. You could just declare X on a function scope and pass it to your function using a lambda:

from itertools import combinations
import numpy as np
from scipy.stats import pearsonr
from multiprocessing import Pool


def function(X, cols):
    result = X[:, cols]
    x,y = result[:,0], result[:,1]
    result = pearsonr(x,y)
    return result


def main():
    X = np.random.random(100000*10).reshape((100000, 10))
    with Pool(processes=4) as P:
        print(P.map(lambda cols: function(X, cols), 
                    combinations(range(X.shape[1]), 2)))


if __name__ == '__main__':
    main()

If you are not going to write anything to X (which it seems you are not doing), and just going to read from it, you should be good to just have all processes access the same variable without some locking mechanism.

Now, global is not necesarily the way to go. Here is different approach:

from itertools import combinations
import numpy as np
from scipy.stats import pearsonr
from multiprocessing import Pool


class MyCalculator:

    def __init__(self, X):
        self.X = X

    def function(self, cols):
        result = self.X[:, cols]
        x,y = result[:,0], result[:,1]
        result = pearsonr(x,y)
        return result


def main():
    X = np.random.random(100000*10).reshape((100000, 10))
    myCalculator = MyCalculator(X)
    with Pool(processes=4) as P:
        print(P.map(myCalculator.function, 
                    combinations(range(X.shape[1]), 2)))


if __name__ == '__main__':
    main()
Source Link
Miguel Alorda
  • 1.4k
  • 7
  • 14

If you are not going to write anything to X (which it seems you are not doing), and just going to read from it, you should be good to just have all processes access the same variable without some locking mechanism.

Now, global is not necesarily the way to go here. You could just declare X on a function scope and pass it to your function using a lambda:

from itertools import combinations
import numpy as np
from scipy.stats import pearsonr
from multiprocessing import Pool


def function(X, cols):
    result = X[:, cols]
    x,y = result[:,0], result[:,1]
    result = pearsonr(x,y)
    return result


def main():
    X = np.random.random(100000*10).reshape((100000, 10))
    with Pool(processes=4) as P:
        print(P.map(lambda cols: function(X, cols), 
                    combinations(range(X.shape[1]), 2)))


if __name__ == '__main__':
    main()