gstreamer custom plugin for nvidia gpu

user_4886927 0
Viewed 3198

I want to develop a gstreamer plugin that can use the acceleration provided by the GPU of the graphics card (NVIDIA RTX2xxx). The objective is to have a fast gstreamer pipeline that process a video stream including on it a custom filter.

After two days googling, I can not find any example or hint.

  1. One of the best alternatives found is use "nvivafilter", passing a cuda module as argument. However, no where explains how to install this plugin or provides an example. Worst, it seems that it could be specific for Nvidia Jetson hardware.

  2. Another alternative seems use gstreamer inside an opencv python script. But that means a mix that I do not known how impacts performance.

  3. This gstreamer tutorial talks about several libraries. But seems outdated and not provides details.

  4. RidgeRun seems to have something similar to "nvivafilter", but not FOSS.

Has someone any example or suggestion about how to proceed.

nvidia gstreamer gpu
1 Answers

I suggest you start with installing DS 5.0 and explore the examples and the apps provided. It's built on Gstreamer. Deepstream Instalation guide

The installation is straight forward. You will find custom parsers built. You will need to install the following: Ubuntu 18.04, GStreamer 1.14.1, NVIDIA driver 440or later,CUDA 10.2,TensorRT 7.0 or later.

Here is an example of running an app with 4 streams. deepstream-app -c /opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt

The advantage of DS is that all the video pipeline is optimized on GPU including decoding and preprocessing. You can always run Gstreamer along opencv only, in my experience it's not an efficient implementation.

Building custom parsers: The parsers are required to convert the raw Tensor data from the inference to (x,y) location of bounding boxes around the detected object. This post-processing algorithm will vary based on the detection architecture. If using Deepstream 4.0, Transfer Learning Toolkit 1.0 and TensorRT 6.0: follow the instructions in the repository https://github.com/NVIDIA-AI-IOT/deepstream_4.x_apps

If using Deepstream 5.0, Transfer Learning Toolkit 2.0 and TensorRT 7.0: keep following the instructions from https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps

Resources:

  1. Starting page: https://developer.nvidia.com/deepstream-sdk
  2. Deepstream download and resources: https://developer.nvidia.com/deepstream-getting-started
  3. Quick start manual: https://docs.nvidia.com/metropolis/deepstream/dev-guide/index.html
  4. Integrate TLT model with Deepstream SDK: https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps
  5. Deepstream Devblog: https://devblogs.nvidia.com/building-iva-apps-using-deepstream-5.0/
  6. Plugin manual: https://docs.nvidia.com/metropolis/deepstream/plugin-manual/index.html
  7. Deepstream 5.0 release notes: https://docs.nvidia.com/metropolis/deepstream/DeepStream_5.0_Release_Notes.pdf
  8. Transfer Learning Toolkit v2.0 Release Notes: https://docs.nvidia.com/metropolis/TLT/tlt-release-notes/index.html
  9. Transfer Learning Toolkit v2.0 Getting Started Guide: https://docs.nvidia.com/metropolis/TLT/tlt-getting-started-guide/index.html
  10. Metropolis documentation: https://docs.nvidia.com/metropolis/
  11. TensorRT: https://developer.nvidia.com/tensorrt
  12. TensorRT documentation: https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html
  13. TensorRT Devblog: https://devblogs.nvidia.com/speeding-up-deep-learning-inference-using-tensorrt/
  14. TensorRT Open Source Software: https://github.com/NVIDIA/TensorRT
  15. https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html?gi-language=cGood luck.
user_118868780
Related