How to terminate a python multiprocess pool early

Viewed 1403

I've recently started poking around at the multiprocessing module and find the pool.map function very useful for parsing a large array very quickly. Is there a way to terminate a pool early however? Lets say I have a huge list and I want to find a number in a list, check if it's devisable by x, and then return true if it is and terminate the rest of the pool early, how might I go about doing this? For a proof of concept, I'm trying to find prime numbers from 3 to infinity (the least efficient way possible). Here's an example:

import multiprocessing
from functools import partial

finders=multiprocessing.pool(multiprocessing.cpu_count()-1)

def is_devis(x, number):
    if number%x==0:
        return True

if __name__=="__main__":
    Primes=[3, 5, 7, 11, 13, 17, ...]
    x=3
    while True:
        x=x+2
        func=partial(is_devis, x)
        results=finders.map(func, Primes)
        if not (True in results):
            Primes.append(x)

I might not completely grasp how multiprocess pools or the pool.map function works but from what I understand, it will split an iterable up evenly for you and then spread them out amongst the pool and the workers will continue until all the processes return or finish. Is there a way to terminate a pool as soon as one process returns a value? I have looked at the documentation on multiprocess.pool but it is noted

Worker processes within a Pool typically live for the complete duration of the Pool’s work queue.

Thanks in advance!

1 Answers

A naive approach would be to have a global flag that a thread can set when it finds an answer. In each of the other threads, you can periodically check the flag and have the thread terminate if it is set.

Related