I was trying to load a compiled torch_tensorRT module on C++ and python.
In C++ I am loading it with:
const torch::Device device = torch::Device(torch::kCUDA, 0);
model = torch::jit::load(model_path, device);
In python I am loading it with:
torch.jit.load(model_path)
On C++ it worked fine on Debug mode, but on C++ Release mode and In Python it throwed following error:
RuntimeError: Unknown type name 'torch.torch.classes.tensorrt.Engine': File "code/torch/movinets/models.py", line 4 parameters = [] buffers = [] _torch___movinets_models_MoViNet_trt_engine : torch.torch.classes.tensorrt.Engine < -- HERE def forward(self_1: torch.movinets.models.MoViNet_trt, input_0: Tensor) -> Tensor:
However when I loaded torchtrt_runtime.dll on C++ release with following code, torch_tensorRT module got loaded perfectly:
HMODULE hLib = LoadLibrary(TEXT("torchtrt_runtime"));
if (hLib == NULL) {
std::cerr << "Library torchtrt_runtime.dll not found" << std::endl;
exit(1);
}
This makes it clear that running on windows need torchtrt_runtime.dll to be loaded. However when I'm trying to load torchtrt_runtime.dll in python with following code:
torch.ops.load_library("torchtrt_runtime.dll")
or
import ctypes
hllDll = ctypes.WinDLL("torchtrt_runtime.dll") or ctypes.CDLL("torchtrt_runtime.dll")
Library is not getting loaded and throwing following error:
hllDll = ctypes.WinDLL ("torchtrt_runtime.dll") File "C:\Users\NomanAnjum\AppData\Local\Programs\Python\Python310\lib\ctypes_init_.py", line 374, in init self._handle = _dlopen(self._name, mode) FileNotFoundError: Could not find module 'torchtrt_runtime.dll' (or one of its dependencies). Try using the full path with constructor syntax.
I then used dependancy checker for torchtrt_runtime.dll and added all dependant dlls to same folder and error has changed to:
File "video_play.py", line 198, in get_torch_tensorrt_converted_model hllDll = ctypes.WinDLL(r"torchtrt_runtime.dll") File "C:\Users\NomanAnjum\AppData\Local\Programs\Python\Python310\lib\ctypes_init_.py", line 374, in init self._handle = _dlopen(self._name, mode) OSError: [WinError 1114] A dynamic link library (DLL) initialization routine failed
I have this strong feeling that if I can load torchtrt_runtime.dll in python, it will run.
Please help me on this