A consistent example for using the C++ API of Pyarrow

user_10090558 0
Viewed 102

I am trying to use the C++ API of Pyarrow. There is currently no example for it on the official documentation, and this is the best I am able to come up with for a simple thing:

#include <arrow/python/pyarrow.h>
#include <arrow/python/platform.h>
#include "arrow/python/init.h"
#include "arrow/python/datetime.h"

#include <iostream>

void MyFunction(PyObject * obj)
{
        Py_Initialize();
        std::cout << Py_IsInitialized() << std::endl;
        int ret = arrow_init_numpy();
        std::cout << ret << std::endl;
        if (ret != 0) {
          throw 0;
        }
        ::arrow::py::internal::InitDatetime();

        if(arrow::py::import_pyarrow() != 0)
         {
                std::cout << "problem initializing pyarrow" << std::endl;
                throw 0;}
        std::cout << "test" << std::endl;
        Py_Finalize();
        //return arrow::py::is_array(obj);
}

I am trying to compile it with

gcc -pthread -B /home/ziheng/anaconda3/envs/da/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O0 -Wall -Wstrict-prototypes -fPIC -D_GLIBCXX_USE_CXX11_ABI=0 -I/home/ziheng/anaconda3/envs/da/lib/python3.7/site-packages/numpy/core/include -I/home/ziheng/anaconda3/envs/da/lib/python3.7/site-packages/pyarrow/include -I/home/ziheng/anaconda3/envs/da/include/python3.7m -c example.cpp -o example.o -std=c++11
g++ -pthread -shared -fPIC -B /home/ziheng/anaconda3/envs/da/compiler_compat -L/home/ziheng/anaconda3/envs/da/lib -Wl,-rpath=/home/ziheng/anaconda3/envs/da/lib -Wl,--no-as-needed -Wl,--sysroot=/ example.o -L/home/ziheng/anaconda3/envs/da/lib/python3.7/site-packages/pyarrow -l:libarrow.so.600 -l:libarrow_python.so.600 -l:libpython3.7m.so -o example.cpython-37m-x86_64-linux-gnu.so

The compilation works with no problems. However when I try to use ctypes to call the compiled .so file, like this:

from ctypes import *
lib = CDLL('example.cpython-37m-x86_64-linux-gnu.so')
lib._Z10MyFunctionP7_object(1)

I get segmentation fault at arrow_init_numpy, after Py_IsInitialized() prints 1. When I run it through gdb, I get/tmp/build/80754af9/python_1614362349910/work/Python/ceval.c: No such file or directory.

If I try to compile my C code as a standalone executable, however, it works with no problems.

Can someone please help? Thank you.

pyarrow
1 Answers

First, the call to Py_Initialize() is superfluous. You are calling your code from within python and so, presumably, python has already been initialized. That would be needed if you were writing your own main and not a plugin-type library. Correspondingly, the call to Py_Finalize() is probably a bad idea.

Second, and more significant for the error at hand, is that you are using ctypes.CDLL (and not, for example, ctypes.PyDLL) which states (emphasis mine):

The returned function prototype creates functions that use the standard C calling convention. The function will release the GIL during the call. If use_errno is set to true, the ctypes private copy of the system errno variable is exchanged with the real errno value before and after the call; use_last_error does the same for the Windows error code.

And, finally, the Arrow initialization routines assume you are holding the GIL (this should probably be added to the documentation). So the easiest way to fix your program is probably to change CDLL to PyDLL:

from ctypes import *
lib = PyDLL('example.cpython-37m-x86_64-linux-gnu.so')
lib._Z10MyFunctionP7_object(1)
user_2026940
Related