Pybind11 accessing on opaque vectors of opaque vectors

user_6614127 0
Viewed 2257

To be able to pass vectors of a custom type by reference between Python and C++, my project uses PYBIND11_MAKE_OPAQUE and pybind11::bind_vector<> on vectors of my type. However, I also need to work with a vector of vector of my custom type. Below is an example.

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <pybind11/stl_bind.h>
#include <vector>

class Example {};

PYBIND11_MAKE_OPAQUE(std::vector<Example>);
PYBIND11_MAKE_OPAQUE(std::vector<std::vector<Example>>);

PYBIND11_MODULE(ExModule, m)
{
    pybind11::class_<Example>(m, "Example")
        .def(pybind11::init<>());

    pybind11::bind_vector<std::vector<Example>>(m, "ExampleVector");
    pybind11::bind_vector<std::vector<std::vector<Example>>>(m, "Example2DVector");
}

If I create a 2D vector of my type in Python, then try to access it, I get an error. Below is the example Python code.

from ExModule import Example, ExampleVector, Example2DVector

# a is a 10x10 vector of Examples
a = Example2DVector([ExampleVector([Example() for i in range(10)]) for i in range(10)])

b = a[4]

Error message:

TypeError: Unable to convert function return value to a Python type! The signature was
    (self: ExModule.Example2DVector, arg0: int) -> std::vector<Example, std::allocator<Example> >

This seems to be because the return type on the index operation on the 2D vector type is an opaque type. What I think should be happening is the return should be constructed into a ExampleVector. I can't do that from Python, it must be done in the module wrapper. Is this a bug or missing feature? If not, how do I overload the index operator on the Example2DVector class?

pybind11 c++
2 Answers

The example works, but my code did not. This was due to the fact that in my code the 1D vector class was bound to a Python type in a different pybind11 module. So there was no mapped Python type for the 1D vector C++ type when it defined the __getitem__ for the 2D vector class. However, I think that if I imported the module containing the 1D vector Python type bind later that it should work, but it does not. That might be a bug.

EDIT:

This behavior is not a bug (I figured, Jakob seems like a pretty smart guy). As discussed in the manual on binding STL containers, there is a section on "module local" bindings. By default type bindings are local to the module they are defined in to avoid multiple different bindings of the same type.

However, our project contains a "datatypes" module, and many modules that use those types. In this case we do not want datatypes defined in the "datatypes" module to be module local. Otherwise we end up with the given problem of return values not being converted into the correct Python type.

We can turn off the default module local binding in the binding definition. Using the question's example, we can turn off the module local binding for ExampleVector and accesses to an Example2DVector (defined in another module) will no longer fail.

pybind11::bind_vector<std::vector<Example>>(m, "ExampleVector", pybind11::module_local(false));
user_66141270

Quote form docs:

This macro must be specified at the top level (and outside of any namespaces), since it instantiates a partial template overload. If your binding code consists of multiple compilation units, it must be present in every file (typically via a common header) preceding any usage of std::vector. Opaque types must also have a corresponding class_ declaration to associate them with a name in Python, and to define a set of available operations

@ktb, It's not a bug, please see https://pybind11.readthedocs.io/en/stable/advanced/cast/stl.html?highlight=compilation%20unit#making-opaque-types

user_17414770
Related