python concurrent.futures thread vs. process question

user_12413078 0
Viewed 56

I am trying to get more familiar with concurrent.futures so I can do parallel processing on some more complex tasks. To learn I am just trying to execute this code in python (spyder interpreter):

import concurrent.futures
import time

start = time.perf_counter()


def do_something(seconds):
    print(f'Sleeping {seconds} second(s)...')
    time.sleep(seconds)
    return f'Done Sleeping...{seconds}'

if __name__ =='__main__':
    with concurrent.futures.ThreadPoolExecutor() as executor:
        secs = [5, 4, 3, 2, 1]
        results = executor.map(do_something, secs)
    
        # for result in results:
        #     print(result)
    
    finish = time.perf_counter()
    
    print(f'Finished in {round(finish-start, 2)} second(s)')

and I get the output I expect:

runfile('D:/untitled1.py', wdir='D:/MarketProject')
Sleeping 5 second(s)...
Sleeping 4 second(s)...
Sleeping 3 second(s)...
Sleeping 2 second(s)...
Sleeping 1 second(s)...
Finished in 5.0 second(s)

But when I change 'concurrent.futures.ThreadPoolExecutor()' to concurrent.futures.ProcessPoolExecutor() all I get is

runcell(0, 'D:/untitled1.py')
Finished in 0.12 second(s)

Any insight into why it doesn't work when trying to use processes instead of threads?

multithreading multiprocessing python
1 Answers

I've encountered this issue before with Jupyter notebooks and Python terminals on Windows. The function that you've defined isn't available to each subprocess, so each subprocess is just immediately dying. The solution is to define the function in a separate file and import it, then try to map with that imported function.

user_105435280
Related