pybind11 memory leak and crashes

Viewed 330

I am facing memory leak and crash issues in pybind11.

I am calling a python function "myfunc" from a python file "mydl.py" that uses Tensorflow Keras deep learning functions, Numpy, and Redis modules using pybind11 in a repeatitive C++ code. The code structure is as follows.

class myclass {
  public:
   myclass() {
     py::initialize_interpreter();

     {
        py::module sys = py::module::import("sys");
        py::module os = py::module::import("os");
        py::str cwd = os.attr("getcwd")();
        py::print("os.cwd: ", cwd);
        py::str bin = cwd + py::str("/../bin");

        // Add bin to sys.path
        py::module site = py::module::import("site");
        site.attr("addsitedir")(bin);
      }
   }
   
   ~myclass() {
      py::finalize_interpreter();
   }

   int callpyfunc(string a1, string a2) {
     int retval;
     {
        py::module mydl = py::module::import("mydl");
        py::object result = mydl.attr("myfunc")(a1, a2);
        retval = result.cast<int>();
     }

    return retval;
   }
}

myclass *mcobj1;

int main() {

  mcobj1 = new myclass();
  int retval;

  while (/* some deep learning condition is not met */) {
     retval = mcobj1->callpyfunc(a1, a2);
  }

  del mcobj1;

}

The memory size of this program goes on increasing consistently to the point of it consuming entire 62 GB RAM and crashing. It seems like Python interpreter is not releasing memory allocated for different objects inside each call to "myfunc" of "mydl.py" even after the call gets done.

Here's what all I have tried with no luck of fixing the issue:

  1. Using scoped interpreter inside callpyfunc instead of doing initialize_interpreter and finalize_interpreter. But in that case the code crashes quietly in the second call to "callpyfunc", the first call goes fine. This is exactly what is mentioned here

  2. Moving initialize_interpreter along with import of modules like "sys", "os" and finalize_interpreter inside callpyfunc. But in that case the code crashes in the second call to "callpyfunc" at line py::module mydl = py::module::import("mydl"); and never reaches finalizing of interpreter.

0 Answers
Related