Numpy threadpool size in main process changes when fork occurs

user_3476768 0
Viewed 64

On importing Numpy, the underlying BLAS library used by Numpy allocates a threadpool. This is expected, and can be observed by simply running:

cat /proc/<my python pid>/status | grep Threads

What is weird is that if my main process ever calls fork(), all these threads vanish in my main process. To be clear, this is the main process (I know forked processes don't retain copies of parent threads).

This behavior is easily reproduced. Exact thread numbers may vary by machine:

import numpy as np  # main process has 36 threads
import os

os.fork() # main process has 1 thread

I would like to understand this behavior and where it is coming from, but I'm having a hard time tracking it down.

I'm using Python 3.6.9 and Numpy 1.18.4.

numpy fork python
1 Answers

After some digging, the reason for the behavior is explained here: https://github.com/xianyi/OpenBLAS/issues/294

Also worth noting that threads are spawned once again if I try to do a computation after the fork() that requires Numpy to use underlying BLAS library.

user_34767680
Related