Catch child process exception in parent process

user_5736696 0
Viewed 303

I am creating multiple processes which are running crawlers separately. I want to ensure that if there is some exception in crawler process, I am able to catch it in Parent Process. Here is the process creation code:

try:
    caching_process = Process(target=run_crawler_process, args=(Config.CRAWLER_NAME, locations,
                                                               city_payloads_map, cycle_count))
    caching_process.start()
except Exception as processException:
    raise processException
python-3.x multiprocessing
1 Answers

You cannot do that with Process objects. The multiprocessing.Pool or the concurrent.futures.ProcessPoolExecutor allow to do that.

pool = multiprocessing.Pool()
task = pool.apply_async(run_crawler_process, (Config.CRAWLER_NAME, locations,c ity_payloads_map, cycle_count))

try:
    task.get()
except Exception as error:  
    print("Error while processing task: %s" % error)
user_17944970
Related