We are trying to upgrade scikit-learn version from 0.24.2 to 1.0.2 (or the latest stable version) Our code is running on EKS pods.
From time to time (it’s not always reproducible), when we call means.predict() the process hangs.
There is no high CPU or memory consumption and we’re not seeing any log messages. It looks like nothing really happens in the pod.
After we generated stack traces of all the threads every minute, we saw that the following thread hangs and the stack trace doesn’t change
Thread 0x00007fe54e06c640 (most recent call first):
File "/buildots/pycode/scripts/threadpoolctl.py", line 688 in _make_controller_from_path
File "/buildots/pycode/scripts/threadpoolctl.py", line 588 in match_library_callback
File "/buildots/pycode/scripts/threadpoolctl.py", line 601 in _find_libraries_with_dl_iterate_phdr
File "/buildots/pycode/scripts/threadpoolctl.py", line 559 in _load_libraries
File "/buildots/pycode/scripts/threadpoolctl.py", line 403 in __init__
File "/usr/local/lib/python3.9/site-packages/sklearn/utils/fixes.py", line 141 in _get_threadpool_controller
File "/usr/local/lib/python3.9/site-packages/sklearn/utils/fixes.py", line 147 in threadpool_limits
File "/usr/local/lib/python3.9/site-packages/sklearn/cluster/_kmeans.py", line 784 in _labels_inertia_threadpool_limit
File "/usr/local/lib/python3.9/site-packages/sklearn/cluster/_kmeans.py", line 1025 in predict
The input to _make_controller_from_path that makes the process hang, changes every time. Paths that worked in one process (e.g. /usr/lib/x86_64-linux-gnu/libssl.so.1.1), caused the thread to hang in another process.
When we initialize the threadpoolctl for sklearn first thing in our process by setting sklearn._sklearn_threadpool_controller = threadpoolctl.ThreadpoolController(), this doesn’t happen (we’ve tested it many times)
This is a possible, but not an elegant workaround, and we very much prefer to understand and fix the root cause.
We load the kmeans model via pickle this way: pickle.load(io_bytes).predict(descriptors)
In light of the stack trace and the workaround above, we don’t think that this is the problem.
Did anyone encounter such a problem and can help us understand the root cause?
Thank you