How can I make persistent changes from Pyhon's side to objects stored in std::vector's as std::shared_ptr's

Viewed 224

Imagine I define two classes (Foo and Bar). Bar stores two Foos in a std::vector as std::shared_ptrs. I want to expose everything to Python by using pybind11. In addition, I want Foo to also support dynamic attributes, which requires using pybind11::dynamic_attr() during the binding stage. Everything works fine as long as I don't try to add dynamic attributes to Foo instances only through the vector that stores them. Since it is not easy to explain my issue in words, here is a MWE:

The pybind11 module is defined in the pyissue.cpp file:

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>

#include <vector>
#include <memory>

namespace py = pybind11;

class Foo {
public:
    Foo() {
        a = 5;
    }

    ~Foo() {

    }

    int a;
};

class Bar {
public:
    Bar() {
        foos.push_back(std::make_shared<Foo>());
        foos.push_back(std::make_shared<Foo>());
    }
    ~Bar() {

    }

    std::vector<std::shared_ptr<Foo>> foos;
};


PYBIND11_MODULE(pybug, m) {
    py::class_<Foo, std::shared_ptr<Foo>> foo(m, "Foo", py::dynamic_attr());
    foo
        .def(py::init<>())
        .def_readwrite("a", &Foo::a);

    py::class_<Bar> bar(m, "Bar");
    bar
        .def(py::init<>())
        .def_readwrite("foos", &Bar::foos);

which can be compiled as follows (at least on my Linux box):

g++ pyissue.cpp -shared --std=c++11 -fPIC `python3 -m pybind11 --includes` -o pyissue`python3-config --extension-suffix`

Now the following python snippet works exactly as intended:

import pyissue

bar = pyissue.Bar()
print(bar.foos[0].a) # prints 5

bar.foos[0].a = 2
print(bar.foos[0].a) # prints 2

myfoo = bar.foos[0]
myfoo.b = 3 # this is a dynamic attribute, it exists only on python's side
print(myfoo.b) # prints 3

However, the next snippet raises an AttributeError: 'pybug.Foo' object has no attribute 'b' exception:

import pyissue

bar = pyissue.Bar()

bar.foos[0].b = 2
print(b.foos[0].b) # here comes the exception

My question boils down to: is there any way to make the last snippet work?

Edit: note that if I explicitly keep a reference to an object, then I can use it without any issues. For instance the following code will work as intended:

import pyissue

bar = pyissue.Bar()

myfoo = bar.foos[0]
bar.foos[0].b = 2
print(b.foos[0].b) # prints 2
1 Answers

Let me answer with bare citations from pybind11 doc since it's quite well explained there.

The major downside of these implicit conversions is that containers must be converted (i.e. copied) on every Python->C++ and C++->Python transition, which can have implications on the program semantics and performance. Please read the next sections for more details and alternative approaches that avoid this.

link: https://pybind11.readthedocs.io/en/stable/advanced/cast/stl.html#stl-containers

By default, classes exported from C++ do not support this and the only writable attributes are the ones explicitly defined using class_::def_readwrite() or class_::def_property().

link: https://pybind11.readthedocs.io/en/stable/classes.html#dynamic-attributes

Edit:

bar.foos returns a copy of the std::vector and it's internals. If you don't store returned python objects, you'll get brand-new python objects at each call.

Yes, python Foo wrappers hold the references (plural!) to the original C++ instance. But dynamic attributes are properties of python wrappers, therefore different python wrappers with same referent does not share dynamic attributes.

Edit 2:

Your latter snippet is a bit puzzling. I guess that some caching on pybind11/cython side might be involved.

Update: Answering the title of the question:

How can I make persistent changes from Pyhon's side to objects stored in std::vector's as std::shared_ptr's

Make changes to C++ instance, not to a python wrapper around it.

Related