pybind11 STL autoconverter breaks std::list pointers

Question

I have a C++ library that manipulates (among other things) a list of wrappers that I have been working on converting to Python using pybind11. The rest of the library operates on a pointer to a list of pointers: std::list*. The problem is that when attempting to autocast a Python list to this C++ list and then initializing a ParamMap, an object that holds the list on the C++ side, the pointers of the list get all messed up. Inspection in GDB reveals that the "next-object pointers" of all the objects are invalid, and this leads to segfaults when traversing the list.

There is no sign of the objects being deallocated on the C++ side, as neither the destructors for the list container ParamMap nor the list objects Symbol are called. I've deduced that the issue might be Python hyperactively deleting objects C++ is still using, but I've tried object terms like py::return_value_policy::reference and py::keep_alive, and they haven't fixed the problem. What is going wrong here? Unfortunately, changing the list type on the C++ side is not an option, but I would really appreciate some help in making this work on the Python side. Thank you!

Here is some minimal reproduction code:

Symbol.hpp

#include 

class Symbol {
private:
        std::string val1;
        int val2;

public:
        Symbol(std::string con1, int con2) : val1(con1), val2(con2) {}
};

ParamMap.hpp

#include 
#include "Symbol.hpp"

class ParamMap {
private:
        std::list* lst;
        int otherData;

public:
        ParamMap(std::list* symbolList, int dat) : lst(symbolList), otherData(dat)  {}
        std::list* getSymbols() { return lst; }
        int getOtherData() { return otherData; }
};

Query.cpp

#include 
#include "ParamMap.hpp"

#include 
#include 
namespace py = pybind11;

void getSymbolListSize(ParamMap* map) {
        std::cout << "Entering query method" << std::endl;
        auto sz = map->getSymbols()->size(); // SEGFAULT OCCURS WHEN GETTING SIZE
        std::cout << "Got size successfully. Size = " << sz << std::endl;
}

PYBIND11_MODULE(list_test, handle) {
        handle.def("getSymbolListSize", &getSymbolListSize);

        py::class_(handle, "ParamMap")
                .def(py::init*, int>(), py::keep_alive<1, 2>())
                .def("getOtherData", &ParamMap::getOtherData)
                .def("getSymbols", &ParamMap::getSymbols);

        py::class_(handle, "Symbol")
                .def(py::init());
}

test.py

import list_test as p

# Creating a list of some random symbols
symbol_list = []
symbol1 = p.Symbol("Hello", 1)
symbol_list.append(symbol1)
symbol2 = p.Symbol("World", 2)
symbol_list.append(symbol2)

# Creating a parammap and passing it the symbol list
pm = p.ParamMap(symbol_list, 71)

print("Symbol list and ParamMap init'd successfully")

# Here, calling Query.cpp's only method
sz = p.getSymbolListSize(pm)
print(sz)

user_5140209 · Answer

I don't know a lot about how pybind11 works its magic and therefore I can't help you understanding what is going on. However, I have the feeling that pybind attempts to build the list even though your code only uses a pointer to the list. If I were you I'd consider this a pybind bug and post it as an issue on their github page.

As per your code, doing something like this seems to work (although it's not very clean):

#include 
#include "Symbol.hpp"

class ParamMap {
private:
    std::list* lst;
    int otherData;

public:
    ParamMap(std::list *symbolList, int dat) : lst(symbolList), otherData(dat)  {
      lst = new std::list;
      for(auto s : *symbolList) {
        lst->push_back(s);
      }
    }

    ~ParamMap() {
      delete lst;
    }

    std::list* getSymbols() { return lst; }
    int getOtherData() { return otherData; }
};

I don't know who's supposed to manage the lifetime of the pointed list, so you may want to remove the destructor in case someone else is supposed to deallocate the list.

pybind11 STL autoconverter breaks std::list pointers

1 Answers