Increase ref_count of pybind11 handle in C++

user_2015882 0
Viewed 654

I have a virtual object that I subclass and implement in python using pybind11, and that object then is used in C++ code but the python object goes out of scope when the python script finished and python garbage collector destroys my object before I can use it in C++ side.

To avoid the destruction I have added a "keep" method to the binding code that increases the ref_count, so that python garbage collector doesn't destroy it.

C++ code for the bindings

VNCS::Real VNCS::py::BlendingField::blending(const VNCS::Point_3 &p) const
{
    std::array<double, 3> pyP{p[0], p[1], p[2]};
    PYBIND11_OVERLOAD_PURE(VNCS::Real,          /* Return type */
                           VNCS::BlendingField, /* Parent class */
                           blending,            /* Name of function in C++ (must match Python name) */
                           pyP                  /* Argument(s) */
    );
}

void VNCS::py::module::blendingField(pybind11::module &m)
{
    pybind11::class_<VNCS::BlendingField, VNCS::py::BlendingField, std::shared_ptr<VNCS::BlendingField>>(
        m, "BlendingField")
        .def(pybind11::init(
            [](pybind11::args &args, pybind11::kwargs &kwargs) { return std::make_shared<VNCS::py::BlendingField>(); }))
        .def("keep", [](pybind11::handle handle) { handle.inc_ref(); });
}

Python code

class BlendingField(PyVNCS.BlendingField):
   def __init__(self, *args, **kwargs):
      PyVNCS.BlendingField.__init__(self, *args, **kwargs)

   def __del__(self):
       print("dtor from python")

   def blending(self, point):
       return point[0]

blendingField = BlendingField()
blendingField.keep()
simCreator = initObject.addObject("SimCreator",
                     name="simCreator",
                     coarseMesh="@lowResTopology",
                     detailMesh="@highResTopology")
simCreator.setBlendingField(blendingField)

This doesn't look very nice, as the keep method there feels like a terrible hack. What would be the correct way to implement this?

pybind11 python
1 Answers

I don't think messing with the ref count is the best solution. If you want to use an object in C++-land, use C++ lifetime management tools.

Pybind integrates lifetime management into the language bindings, you just need to change it's behavior. To quote the docs:

The binding generator for classes, class_, can be passed a template type that denotes a special holder type that is used to manage references to the object. If no such holder type template argument is given, the default for a type named Type is std::unique_ptr<Type>, which means that the object is deallocated when Python’s reference count goes to zero.

It is possible to switch to other types of reference counting wrappers or smart pointers, which is useful in codebases that rely on them. For instance, the following snippet causes std::shared_ptr to be used instead.

Here's an example lifted from the tests:

// Object managed by a std::shared_ptr<>
class MyObject2 {
public:
    MyObject2(const MyObject2 &) = default;
    MyObject2(int value) : value(value) { print_created(this, toString()); }
    std::string toString() const { return "MyObject2[" + std::to_string(value) + "]"; }
    virtual ~MyObject2() { print_destroyed(this); }
private:
    int value;
};
py::class_<MyObject2, std::shared_ptr<MyObject2>>(m, "MyObject2")
    .def(py::init<int>());
m.def("make_myobject2_1", []() { return new MyObject2(6); });
m.def("make_myobject2_2", []() { return std::make_shared<MyObject2>(7); });
m.def("print_myobject2_1", [](const MyObject2 *obj) { py::print(obj->toString()); });
m.def("print_myobject2_2", [](std::shared_ptr<MyObject2> obj) { py::print(obj->toString()); });
m.def("print_myobject2_3", [](const std::shared_ptr<MyObject2> &obj) { py::print(obj->toString()); });
m.def("print_myobject2_4", [](const std::shared_ptr<MyObject2> *obj) { py::print((*obj)->toString()); });

If you do go that route, which I would recommend, this question would be a duplicate of this one.

Note that there are some minor pitfalls here especially if youre codebase uses raw pointers around python code. The docs point to a possible double-free if a C++ method returning a raw pointer to an shared object is called from python. Again stealing from the docs:

class Child { };

class Parent {
public:
   Parent() : child(std::make_shared<Child>()) { }
   Child *get_child() { return child.get(); }  /* Hint: ** DON'T DO THIS ** */
private:
    std::shared_ptr<Child> child;
};

PYBIND11_MODULE(example, m) {
    py::class_<Child, std::shared_ptr<Child>>(m, "Child");

    py::class_<Parent, std::shared_ptr<Parent>>(m, "Parent")
       .def(py::init<>())
       .def("get_child", &Parent::get_child);
}

from example import Parent
print(Parent().get_child()) // UB or Segfault

Though these issue don't seem to me to be unique to shared_ptr over unique_ptr.

user_32087230
Related