Python 3.8 concurrent.futures "OSError: handle is closed"

Question

I have Python 3.8 and cannot upgrade (I have dependencies that only work with Python 3.8), so this SO post does not answer my question. How do I prevent the OSError: handle is closed error? This happens when returning a result to the main thread. Here is a very trivial example:

from concurrent.futures import ProcessPoolExecutor


def my_func(num):
    return num + 1


def worker(data):
    with ProcessPoolExecutor(5) as executor:
        result = executor.map(my_func, data)
    return result


if __name__ == "__main__":
    arr = [1, 2, 3, 4, 5]
    print(arr)
    arr = worker(arr)
    print(tuple(arr))

This gives

[1, 2, 3, 4, 5]
(2, 3, 4, 5, 6)
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "C:\Users\hendra11\AppData\Local\Programs\Python\Python38\lib\concurrent\futures\process.py", line 102, in _python_exit
    thread_wakeup.wakeup()
  File "C:\Users\hendra11\AppData\Local\Programs\Python\Python38\lib\concurrent\futures\process.py", line 90, in wakeup
    self._writer.send_bytes(b"")
  File "C:\Users\hendra11\AppData\Local\Programs\Python\Python38\lib\multiprocessing\connection.py", line 183, in send_bytes
    self._check_closed()
  File "C:\Users\hendra11\AppData\Local\Programs\Python\Python38\lib\multiprocessing\connection.py", line 136, in _check_closed
    raise OSError("handle is closed")
OSError: handle is closed

Does anyone know what's causing this bug and how I can manually fix it?

If it matters, I'm on Windows 10 running Visual Studio Code.

UPDATE: This error goes away when I use ThreadPoolExecutor AND it is much faster, but I don't know why.

Python 3.8 concurrent.futures "OSError: handle is closed"

0 Answers