I am facing memory leak and crash issues in pybind11.
I am calling a python function "myfunc" from a python file "mydl.py" that uses Tensorflow Keras deep learning functions, Numpy, and Redis modules using pybind11 in a repeatitive C++ code. The code structure is as follows.
class myclass {
public:
myclass() {
py::initialize_interpreter();
{
py::module sys = py::module::import("sys");
py::module os = py::module::import("os");
py::str cwd = os.attr("getcwd")();
py::print("os.cwd: ", cwd);
py::str bin = cwd + py::str("/../bin");
// Add bin to sys.path
py::module site = py::module::import("site");
site.attr("addsitedir")(bin);
}
}
~myclass() {
py::finalize_interpreter();
}
int callpyfunc(string a1, string a2) {
int retval;
{
py::module mydl = py::module::import("mydl");
py::object result = mydl.attr("myfunc")(a1, a2);
retval = result.cast<int>();
}
return retval;
}
}
myclass *mcobj1;
int main() {
mcobj1 = new myclass();
int retval;
while (/* some deep learning condition is not met */) {
retval = mcobj1->callpyfunc(a1, a2);
}
del mcobj1;
}
The memory size of this program goes on increasing consistently to the point of it consuming entire 62 GB RAM and crashing. It seems like Python interpreter is not releasing memory allocated for different objects inside each call to "myfunc" of "mydl.py" even after the call gets done.
Here's what all I have tried with no luck of fixing the issue:
Using scoped interpreter inside
callpyfuncinstead of doinginitialize_interpreterandfinalize_interpreter. But in that case the code crashes quietly in the second call to "callpyfunc", the first call goes fine. This is exactly what is mentioned hereMoving
initialize_interpreteralong with import of modules like "sys", "os" andfinalize_interpreterinsidecallpyfunc. But in that case the code crashes in the second call to "callpyfunc" at linepy::module mydl = py::module::import("mydl");and never reaches finalizing of interpreter.