Aggregating complex results from multiple processes?

Viewed 40

Set Up

I have a complex set of data that I need to compare to various sets of constraints concurrently, but I'm running into multiple issues. The first issue is getting results out of my multiple processes, and the second issue is making anything beyond an extremely simple function to run concurrently

Part two of this is a bit more complicated
Link to part Two: Click Here for Part Two

Example:

I have a simple function that completes an action but I can't access the results outside of the processes that are spawned in.

Functions

Because I'm using a Jupyter notebook on windows I have the functions saved in a separate file (multi_process_functions.py) and I'm importing that file as mpf

def f2(name):
    return('hello ' + name)

def f3(name):
    return('Bye ' + name)

Jupyter Notebook

Then in my jupyter notebook I have the following code:

data_manager = mp.Manager()
ns = data_manager.Namespace()
ns.names = ['Bob', 'Joe', 'Brian']
ns.results = []

if __name__ == '__main__':
    with mp.Pool() as p:
        res = p.map(mpf.f2, ns.names)
        ns.results.append(res)
        print(res)
    with mp.Pool() as x:
        res = x.map(mpf.f3, ns.names)
        ns.results.append(res)
        print(res)
    
    print(ns.results)

The resulting output:

['hello Bob', 'hello Joe', 'hello Brian']
['Bye Bob', 'Bye Joe', 'Bye Brian']
[]

I was hoping to have the output look like this:

['hello Bob', 'hello Joe', 'hello Brian']
['Bye Bob', 'Bye Joe', 'Bye Brian']
[['hello Bob', 'hello Joe', 'hello Brian'], ['Bye Bob', 'Bye Joe', 'Bye Brian'] ]

Question:

How can I aggregate and access the results once all of my processes have finished?

0 Answers
Related