Multi Processing with python: saving objects of the "pooled" functions

user_4566565 0
Viewed 86

just wanted to know a way to save the objects that come out of the functions that I pool in a multi-process environment (python 3.5).

Code to pool (a simple example):

import numpy as np

vct_list = [0, 1]

def f_to_pool(vector_list):
    chosen_nmb = np.random.choice(vector_list)
    status_ = True if chosen_nmb == 0 else False
    return chosen_nmb * 4, status_

scenarios = range(100)

numbers_processed = []
all_status = []

for _ in scenarios:
    nmb_processed, status = f_to_pool(vct_list)
    numbers_processed.append(nmb_processed)
    all_status.append(status)

whereas the multi-processing framework is multiprocessing.Pool:

pool = multiprocessing.Pool(nmb_of_proc)
results = pool.map(function, objects)
pool.close()

the issue is that, even if I could run the in async and collect the results as shown in the docs ( https://docs.python.org/3.5/library/multiprocessing.html#using-a-pool-of-workers), the process would remain the same and, since I am using this technique to expand my computational power (this is NOT an I/O problem), that would be of no use to me.

Any suggestions?

multiprocessing python
1 Answers

Your research is correct, you may combine both things:

import multiprocessing

def your_time_consuming_function(i):
    status = bool(i % 2)
    return i * 4, status

def run(scenarios_list):
    with multiprocessing.Pool(4) as pool:
        return pool.map(your_time_consuming_function, scenarios_list)

scenarios = range(8)

print(run(scenarios))

Then will product such result:

[(0, False), (4, True), (8, False), (12, True), (16, False), (20, True), (24, False), (28, True)]
user_11790090
Related