How to get precise timeout with Python multiprocess apply_async?

user_5115437 0
Viewed 2977

I'm using multiprocess pool to run a batch of jobs in parallel. I want to give each job a 2 second time limit. That is, I want to kill a job if it is taking longer than 2 seconds and move on to the next job.

Here is my code

from multiprocessing import Pool, TimeoutError                                                                                                 
import time                                                                                                                                    
import os                                                                                                                                      
import random                                                                                                                                  

def f():                                                                                                                                                                                                                                                           
    time.sleep(3)
    return True                                                                                                                            

if __name__ == '__main__':                                                                                                                                                                                                            
    with Pool(processes=8) as pool:
    multiple_results = [pool.apply_async(f, ()) for i in range(16)]
    for res in multiple_results:                                                                                                           
        try:                                                                                                                               
            print (res.get(timeout=2)),                                                                                                   
        except:                                                                                                                            
            print ("TO"),

I would expect 16 TOs to be printed, however, what I get is TO True True True True... This means that most execution of the function f is successful. I didn't expect this since I thought I set a 2 seconds timeout for each spawned process. Am I having some misunderstanding of the usage of the apply_async function? Or is there a better way to achieve what I want to do?

python-pool python-multiprocessing python
1 Answers

The timeout on the AsyncResult.get function does not terminate the ongoing job. It just blocks up until timeout seconds and then raises TimeoutError allowing the main thread to resume its execution.

Therefore, all the other enqueued jobs are correctly processed in the background while you wait on the first result and hence you observe such behaviour.

If you want the job to be stopped on timeout you need to rely on third party Pool implementations such as billiard or pebble.

user_17944970
Related