Joblib not working `RLIMIT_NPROC -1 current, -1 max`

Question

I have been stuck with joblib for several days. I am trying to run in parallel some short OpenCV image processing pipeline. Apparently, there is some conflict I am unaware of and neither can I google anything about it. Unfortunately, I have troubles designing proper minimal reproducible example, but maybe someone would help me based on the error messages.

I was first getting this error message (despite the error message, the code worked):

[ERROR:0@3.942] global /io/opencv/modules/core/src/parallel_impl.cpp (240) WorkerThread 43: Can't spawn new thread: res = 11

Now, the following error has added and the code does not run at all, with maximum number of allowed processes somehow being -1:

OpenBLAS blas_thread_init: pthread_create failed for thread 38 of 45: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max

I've tried to set the number of OpenCV threads to 1 (cv2.setNumThreads(1)) so that the parallelization happens only on the side of joblib but that only changed the error:

  File "/root/.cache/pypoetry/virtualenvs/env-py3.7/lib/python3.7/site-packages/joblib/parallel.py", line 968, in __call__
    n_jobs = self._initialize_backend()
  File "/root/.cache/pypoetry/virtualenvs/env-py3.7/lib/python3.7/site-packages/joblib/parallel.py", line 736, in _initialize_backend
    **self._backend_args)
  File "/root/.cache/pypoetry/virtualenvs/env-py3.7/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 470, in configure
    self._pool = MemmappingPool(n_jobs, **memmappingpool_args)
  File "/root/.cache/pypoetry/virtualenvs/env-py3.7/lib/python3.7/site-packages/joblib/pool.py", line 323, in __init__
    super(MemmappingPool, self).__init__(**poolargs)
  File "/root/.cache/pypoetry/virtualenvs/env-py3.7/lib/python3.7/site-packages/joblib/pool.py", line 205, in __init__
    super(PicklingPool, self).__init__(**poolargs)
  File "/root/.pyenv/versions/3.7.12/lib/python3.7/multiprocessing/pool.py", line 176, in __init__
    self._repopulate_pool()
  File "/root/.pyenv/versions/3.7.12/lib/python3.7/multiprocessing/pool.py", line 241, in _repopulate_pool
    w.start()
  File "/root/.pyenv/versions/3.7.12/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/root/.pyenv/versions/3.7.12/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/root/.pyenv/versions/3.7.12/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/root/.pyenv/versions/3.7.12/lib/python3.7/multiprocessing/popen_fork.py", line 70, in _launch
    self.pid = os.fork()
BlockingIOError: [Errno 11] Resource temporarily unavailable

What is wrong with using OpenCV with joblib? Can it be done?

Joblib not working `RLIMIT_NPROC -1 current, -1 max`

0 Answers