Alternative Use Patterns For Python Multiprocessing Avoiding Proliferation Of Global State?

This (enormously simplified example) works fine (Python 2.6.6, Debian Squeeze): from multiprocessing import Pool import numpy as np src=None def process(row): return np.sum(s

Solution 1:

You could always pass a callable object like this, then the object can containe the shared state:

from multiprocessing import Pool
import numpy as np

class RowProcessor(object):
    def __init__(self, src):
        self.__src = src
    def __call__(self, row):
        return np.sum(self.__src[row])

def main():
    src=np.ones((100,100))
    p = RowProcessor(src)

    pool=Pool(processes=16)
    rows = pool.map(p, range(100))
    print rows

if __name__ == "__main__":
    main()