Passing sympy lambda to multiprocessing.Pool.map

user_5133167 0
Viewed 1050

I want to execute a sympy lambda function in parallel. I don't know:

  • why it works in parallel although it is a lambda function
  • why it stops working when I try executing without the pool
  • why it works if I uncomment the first return in lambdify

And apparently the markdown preprocessor needs a line of text above the code so this is the code: 

from multiprocessing import Pool

import sympy
from sympy.abc import x

def f(m):
    return m.lambdify()(1)

class Mult():
    def lambdify(self):
        # return sympy.lambdify(x, 2*x, 'numpy')
        self._lambdify = sympy.lambdify(x, 2 * x, 'numpy')
        return self._lambdify

if __name__ == '__main__':
    with Pool() as pool:
        m = Mult()
        print(pool.map(f, [m]))
        print(pool.map(f, [m]))
        print(f(m))
        print(pool.map(f, [m]))

It prints:

[2]
[2]
2
PicklingError: Can't pickle <function <lambda> at 0x000000000DF0D048>: attribute lookup <lambda> on numpy failed

(I cut the traceback)

If I uncomment, it works normally:

[2]
[2]
2
[2]

I tested only on Windows and it works exactly the same with 'numexpr' instead of 'numpy'.

sympy multiprocessing python
2 Answers

Though I have not fully explored this yet, I just want to put on record that the same example works just fine when using loky instead of multiprocessing:

from loky import get_reusable_executor

import sympy
from sympy.abc import x

def f(m):
    return m.lambdify()(1)

class Mult():
    def lambdify(self):
#        return sympy.lambdify(x, 2*x, 'numpy')
        self._lambdify = sympy.lambdify(x, 2 * x, 'numpy')
        return self._lambdify


executor = get_reusable_executor()

m = Mult()
print('pool.map(f, [m])', list(executor.map(f, [m])))
print('pool.map(f, [m])', list(executor.map(f, [m])))
print('f(m)', f(m))
print('pool.map(f, [m])', list(executor.map(f, [m])))

with output

pool.map(f, [m]) [2]
pool.map(f, [m]) [2]
f(m) 2
pool.map(f, [m]) [2]
user_100202830
Related