Pass kwargs to starmap while using Pool in Python

user_3792469 0
Viewed 4263

I'm using Pool to multithread my programme using starmap to pass arguments.

I'm stuck because I cannot seem to find a way to pass kwargs along with the zip arrays that I'm passing in the starmap function.

pool = Pool(NO_OF_PROCESSES)
branches = pool.starmap(fetch_api, zip(repeat(project_name), api_extensions))

The branches request is incomplete as I'm still not able to figure out how to pass keywords arguments.

def fetch_api(project_name, api_extension, payload={}, headers={}, API_LINK=API_LINK, key=False):
    headers[AUTH_STRING] = 'Gogo'
    call_api = API_LINK + project_name + api_extension
    response_api = requests.get(call_api, headers=headers, params=payload)

    if key: return project_name + ':' + response_api
    else: return response_api

While calling fetch_api() from the branch line, I want to pass payload as {'a':1} and key=True.

Please guide me to the direction or answer. Thanks. Using Python 3.3+.

python-multiprocessing threadpool multithreading python
1 Answers

You can create a wrapper around pool.starmap that also accepts an iterator of over kwargs dictionaries.

from itertools import repeat

def starmap_with_kwargs(pool, fn, args_iter, kwargs_iter):
    args_for_starmap = zip(repeat(fn), args_iter, kwargs_iter)
    return pool.starmap(apply_args_and_kwargs, args_for_starmap)

def apply_args_and_kwargs(fn, args, kwargs):
    return fn(*args, **kwargs)

Then you can call it in your case as:

args_iter = zip(repeat(project_name), api_extensions)
kwargs_iter = repeat(dict(payload={'a': 1}, key=True))
branches = starmap_with_kwargs(pool, fetch_api, args_iter, kwargs_iter)
user_19464650
Related