Freeze/Fail when using functional with OpenMP [Pybind11/OpenMP]

user_17167772 0
Viewed 130

I have a problem with the functional feature of Pybind11 when I use it with a for-loop with OpenMP. I've done some research and my problem sounds pretty similar to the one in this Pull Request from 2 years ago, but although this PR is closed and the issue seems to be fixed I still have this issue. A code example I created will hopefully explain my problem better:

b.h

#include <pybind11/pybind11.h>
#include <pybind11/functional.h>
#include <omp.h>

namespace py = pybind11;

class B {
  public:
    B(int n, const int& initial_value);
    void map(const std::function<int(int)> &f);
  private:
    int n;
    int* elements;
};

b.cpp

#include <pybind11/pybind11.h>
#include <pybind11/functional.h>
#include "b.h"

namespace py = pybind11;

B::B(int n, const int& v)
    : n(n) {
  elements = new int[n];
  #pragma omp parallel for
  for (int i = 0; i < n; i++) {
    elements[i] = v;
  }
}

void B::map(const std::function<int(int)> &f) {
  #pragma omp parallel for
  for (int i = 0; i < n; i++) {
    elements[i] = f(elements[i]);
  }
}

PYBIND11_MODULE(m, handle) {
  handle.doc() = "Example Module";
  py::class_<B>(handle, "B")
  .def(py::init<int, int>())
  .def("map", &B::map)
  ;
}

CMakeLists.txt

cmake_minimum_required(VERSION 3.4...3.18)
project(example)

find_package(OpenMP)

add_subdirectory(pybind11)
pybind11_add_module(m b.cpp)

if(OpenMP_CXX_FOUND)
  target_link_libraries(m PUBLIC OpenMP::OpenMP_CXX)
else()
  message( FATAL_ERROR "Your compiler does not support OpenMP" )
endif()

test.py

from build.m import *


def test(i):
    return i * 20


b = B(2, 2)
b.map(test)

I basically have an array where I want to apply a Python function to every element using a for-loop. I know that it is an issue with functional and OpenMP specifically because in other parts of my project I am using OpenMP successfully and functional is also working if I am not using OpenMP.

Edit: It freezes at the map function and has to be terminated. I am using Ubuntu 21.10, Python 3.9, GCC 11.2.0, OpenMP 4.5, and the newest version of the pybind11 repo.

pybind11 openmp python c++
1 Answers

You're likely experiencing a deadlock between OpenMP's scheduler and Python's GIL (Global Interpreter Lock).

I suggest attaching gdb to your process and looking at where the threads are to verify that's really the problem.

IMHO mixing Python functions and OpenMP like that is asking for trouble. If you want multi-threading of Python functions you can use multiprocessing.pool.ThreadPool. But unless your functions release the GIL most of the time you won't benefit from multi-threading.

user_12768160
Related