Speeding up autowrap from Sympy

Question

I'm trying to convert a large number of Sympy expressions using the autowrap function. It is however a slow process. I've tried parallelizing with some Python packages, but I keep getting errors. My question is, is it possible to speed up the autowrapping process? Like with parallelization or something?

If it is necessary, I can post parts of my script to show how my particular case can be solved, but I'd like to try something more general.

EDIT: The way my code works is (probably not the best way) it creates some expressions from a general expression and its derivatives (Euler-Lagrange equations for those interested). Said expressions are then stored in an array (list of lists) from where they are being called to be autowrapped. Once that is done, they are saved in another array.

This I can do with a simple for loop. It is very very slow for when I have, let's say 400 (aka 20^2) of those expressions.

I've tried using ProcessPoolExecutor from concurrent.futures because I'm already using it in another part of my code. But that throws a BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. error, unless I try to wrap only one function. Then it works fine.

ThreadPoolExecutor does the same thing (or it complains about a wrapper_module_# missing which sometimes is on the tempdir I indicated).

From multiprocessing.Pool I only get Exception in thread Thread-#: and then it keeps running forever unless I stop it. Then it says ModuleNotFoundError: No module named 'wrapper_module_#'.

So I'm not sure why those modules are missing when they are really not.

I'll try to shorten my code as much as possible and I'll post it in a few hours.

EDIT 2: So here's a piece of code that throws the same or similar errors.

# This is the setup (declaring equations and such)
import sympy as sp
from sympy.utilities.autowrap import autowrap
import itertools
import os # <- To change directories because it appears that it changes automatically when using Process and ThreadPoolExecutor
from concurrent.futures import ProcessPoolExecutor,ThreadPoolExecutor

cwd = os.getcwd()
x,y,dx,dy = sp.var('x y dx dy')

eq = list()
eq.append(sp.cos(x)*dx + y*dy) # <- Some random equations
eq.append(3*x*dx + dy)         #    equations as examples.

coeff1 = [eq[0].coeff(dx),eq[0].coeff(dy)] # <- Coefficients of eq[0]
coeff2 = [eq[1].coeff(dx),eq[1].coeff(dy)] # <- Coefficients of eq[1]
coeff = [coeff1,coeff2] # <- This is my list of lists of coefficients
                        #    (currently Sympy expressions)
arguments = [x,y] # <- Arguments for my equations.

Here comes the fun part:

Using list comprehension to call autowrap for every entry of my coeff array works just fine.

coeffComp = [[autowrap( coeff[i][j] , backend='cython', tempdir=cwd+'/CoefficientsComprehension',args=arguments) for j in range(2)] for i in range(2)]

# To visualize:
print(coeff)
# returns [[cos(x), y], [3*x, 1]]

# If I evaluate in (x=1,y=0)
print([[coeffComp[i][j](1,0) for j in range(2)] for i in range(2)])
# returns [[0.5403023058681398, 0.0], [3.0, 1]], which is correct.

However, trying to work with ProcessPoolExecutor or ThreadPoolExecutor

def AutoWrapCoefficients(Index): # <- Function to map
    i,j = Index
    return autowrap( coeff[i][j] , backend='cython', tempdir='./CoefficientsPoolExecutor',args=arguments)


def ToPyOfArray(): # <- Mapping function
    TempArr = list()
    with ThreadPoolExecutor() as executor:
        results = executor.map(AutoWrapCoefficients, itertools.product(range(2),repeat=2))
        for res in results:
            os.chdir(cwd)
            TempArr.append(res)
    return TempArr


if __name__ == '__main__':
    coeffFunctions = ToPyOfArray() # <- Throws described errors

And with multiprocessing:

from multiprocessing import Pool

if __name__ == '__main__':
    with Pool() as p:
        p.map( AutoWrapCoefficients, itertools.product(range(2),repeat=2)) # <- Throws described errors

Speeding up autowrap from Sympy

0 Answers