Problems building a Python/C++ pybind11 package with `CBUILDWHEEL` using Github actions?

Question

I'm currently developing a C++/Python package using pybind11 for the Python bindings. This project is mixed: it has parts which are written in Python and other parts which are written in C++ and are compiled as an external module.

The project uses cmake>1.7. Roughly this is the directory structure of the project:

PythonProject
 -> python_sources
 -> include [cpp headers]
 -> src [cpp sources]
   --> module.cpp [pybind11 bindings declaration]
   --> cpp_sources [where the implementation of the headers goes]

The project also depends on Eigen3.

I have the project setup, so that when building the wheels the cpp module is compiled first and then the *.so (if macOS) is copied to python_sources.

In my machine (macOS 11.2 with XCODE 12) I can generate the wheels with

python -m build or python -m build --sdist(depending if I am only building source distributions)

This works well and it copies the cpp built module correctly to PythonProject/python_sources so that the module can be imported as import python_sources.module.

Also, I'm able to test the installation with pip install -e ./ and all tests pass.

The issue is that when running cbuildwheel on with Github actions, it seems that the c++ module is never built.

I have tried to force building the module and copying it to python_sources, but this also does not work, and the tests do not pass.

This is my cbuildwheel configuration:

- uses: joerick/cibuildwheel@v1.10.0
    env:
      # Python 2.7 on Windows with workaround not supported by scikit-build
      CIBW_SKIP: cp27-win*
      CIBW_BUILD: cp38-${{ matrix.cibw-arch }}
      CIBW_ENVIRONMENT: PYTHONPROJECT_EXTRA_CMAKE_ARGS="-DPREBUILT_DEPENDENCIES=$(python -c 'import os; print(os.getcwd().replace(os.path.sep, '/'))')/build_dependencies"
                        CMAKE_BUILD_PARALLEL_LEVEL=2
                        CMAKE_OSX_ARCHITECTURES="arm64"
      CIBW_BEFORE_ALL_MACOS: |
        brew install libomp && brew install eigen && brew install openblas
      CIBW_BEFORE_ALL_LINUX: |
        yum install eigen3-devel
      CIBW_BEFORE_BUILD: cmake -S . -B build_dependencies $CMAKE_ARCH && cmake --build build_dependencies --target numerical -j 2 && cp build_dependencies/src/*.so python_sources/
      CIBW_BEFORE_BUILD_MACOS: cmake -S . -B build_dependencies -DCMAKE_OSX_ARCHITECTURES="arm64" $CMAKE_ARCH && cmake --build build_dependencies --target module -j 2 && cp build_dependencies/src/*.so python_sources/
      CIBW_BEFORE_TEST: pip install --upgrade -r tests/requirements.txt
      CIBW_TEST_COMMAND: pytest {project}/tests

I want to be able to build these wheels on GitHub actions (not necessarily with cibuildwheels, although that is my preferred option). But I am quite lost here and would really appreciate some help on how to setup the CI configuration.

Problems building a Python/C++ pybind11 package with `CBUILDWHEEL` using Github actions?

0 Answers