Worker pool data structure

user_9451157 0
Viewed 178

I'm wondering whether there's a native implementation in the multiprocessing module that would allow me to store running processes in a list-based structure and whenever a processes is finished with execution It's automatically removed from the list.

In code It'd look like this:

from multiprocessing import process

pool = [] # This data structure needs to prune non-running processes

class A(Process):
     def run():
         pass

for i in range(0, 10):
    worker = A().start()
    pool.append(worker)


# So if I want to iterate the pool now, It should only contain the alive processes

Another way to manage this would be to keep a dictionary:

pool = {
    processId: processObject
}

And then get the active process ids using psutil:

current_process = psutil.Process()
children = current_process.children(recursive=False)

However, what'd be the size of the object inside the dictionary once the process dies?

multiprocess python-multiprocessing python-3.x multiprocessing python
1 Answers

I don't think such a hypothetical self-updating structure would be a good idea, for the same reason you shouldn't modify a list while you're iterating over it. Processes might get removed while you are iterating over the pool.

To iterate over it safely, you would need a snapshot and this would render the whole effort of such a structure pointless. When you need to update your pool-list, you better do so explicitly with for example:

pool[:] = [p for p in pool if p.is_alive()] # p are your processes

or if you want all process-wide, active child-processes and not just that in your custom pool:

[p for p in multiprocessing.active_children()]

You can of course put that somewhere in a function or a method and call it whenever you need an actual pool-list. Processes have a pid attribute so you wouldn't need psutil just for getting process ids.

user_90594200
Related