Hello I have a multiprocessing program like
#this is pseudocode-ish
def worker(queue, context):
set_context(context) #set global context within worker
while queue.qsize() > 0:
process(queue.get(False))
pool = multiprocessing.Pool(20, worker, (queue, global_context))
pool.close()
pool.join()
The problem is that global context is a very heavy object so spawning each individual process (pickling/unpickling) takes a while. So what I've been finding is that for shorter queues, the whole queue gets processed by the first couple of spawned processes and then the rest of the program is stuck spawning the rest of the processes, which inevitably do nothing because there is nothing left in the queue. E.g. each process takes 1 second to spawn, but the queue is processed in 2 seconds - so first two processes finish the queue in 2-3 seconds and then the rest of the program takes 17 seconds to spawn the rest of the queues.
Is there a way to kill the rest of the processes when the queue is empty? Or a more flexible way to set up the pool number of processes - e.g. only spawn another process when needed?
Thanks