parallelize function which has multiple inputs

Admin 1
Viewed 2044

I would like to excecute a function multiple times on different threads with different function inputs.

The problem is, that my function needs for every run more than just one input, so I am trying to do the following: (the function is much simplified here)

import time 
from joblib import Parallel, delayed 

# A function that can be called to do work:
def work(arg1,arg2):
    return (arg1, arg2)

# List of arguments to pass to work():
arg_instances = [(1, 1), (1, 2), (1, 3), (1, 4)]

# Anything returned by work() can be stored:
results = Parallel(n_jobs=4, verbose=1, backend="threading")(map(delayed(work), arg_instances))

print(results)

Unfortunately it does not work.

work() missing 1 required positional argument: 'arg2'

It has something to do with the map function, but I don't know how to solve that. Any help would be much appreciated.

parallel-processing python
1 Answers

You can use partial to create a function callable later.

For example:

funcs = [partial(work, arg1, arg2) for arg1, arg2 in arg_instances]
Parallel(...)(delayed(f) for f in funcs))

Partial is a very handy and useful python builtin function, give it a shot!

Anyway looking at the library (joblib) you are using, probably this will fix

Parallel(...)(delayed(work)(arg1, arg2) for arg1, arg2 in arg_instances))

Actually it is not working because you are calling "work" function with the entire "arg_instances" list as first positional parameter.

user_80785890
Related