RTP Extension Headers over UDP

Viewed 329

I'm trying to send images with certain metadata over UDP using gstreamer. The sender process grabs images from camera, and sends them in the pipeline as appsrc ! rtpvrawpay ! udpsink (the full pipeline string below, in case it's important). And on the reciever side I use udpsrc ! rtpvrawdepay ! appsink.

In order to add metadata I put a probe on the Source Pad of rtpvrawpay, and to read - probe on Sink Pad of rtpvrawdepay. The problems I'm having so far:

  1. The "source" probe receives GST_PAD_PROBE_TYPE_BUFFER_LIST data, while the "sink" probe - GST_PAD_PROBE_TYPE_BUFFER. I would expect that I will get the same data type on both sides of UDP, so why it's not?
  2. I fail to read the twobytes_header that was added by the sender, it fails already when trying to gst_rtp_buffer_map the buffer in the probe. UPDATE: succeeded to read the header on the receiver side, see the code below. Still would like to understand why I see Buffer List VS Buffer on the two probe locations.

Will appreciate any insights, or maybe a better direction for sending metadata with image over UDP.

Sender pipeline: appsrc ! videoconvert ! queue ! rtpvrawpay name=ToRTP ! queue ! udpsink host=192.168.42.49 port=5566 sync=false async=false

Reciever pipeline: udpsrc port=5566 caps="application/x-rtp, media=(string)video, sampling=(string)BGR, clock-rate=(int)90000, depth=(string)8, width=(string)640, height=(string)480, encoding-name=(string)RAW" ! rtpvrawdepay name=FromRTP ! queue ! videoconvert ! appsink

UPDATE: The Code that worked for me:

Sender Code:

GstRTPBuffer rtp_buf = GST_RTP_BUFFER_INIT;
gst_rtp_buffer_map(*buffer, GST_MAP_READWRITE, &rtp_buf);
gst_rtp_buffer_add_extension_twobytes_header(&rtp_buf, 1, 1, myInfo->buf, sizeof(myInfo->buf));
gst_rtp_buffer_unmap(&rtp_buf);

Receiver Code:

GstRTPBuffer rtp_buf = GST_RTP_BUFFER_INIT;
GstBuffer* buf = GST_PAD_PROBE_INFO_BUFFER(info);
gst_rtp_buffer_map(buf, GST_MAP_READ, &rtp_buf);
gpointer myInfoBuf = nullptr;
guint size         = 0;
gst_rtp_buffer_get_extension_twobytes_header(&rtp_buf, &appbits, 1, 0, &myInfoBuf, &size);
gst_rtp_buffer_unmap(&rtp_buf);
1 Answers

The "source" probe receives GST_PAD_PROBE_TYPE_BUFFER_LIST data, while the "sink" probe - GST_PAD_PROBE_TYPE_BUFFER. I would expect that I will get the same data type on both sides of UDP, so why it's not?

On the sender this depends on the type of data that passes through the RTP payloader: small packets (e.g. audio or small video frames) that fit in the MTU configuration of the payloader will trigger the GST_PAD_PROBE_TYPE_BUFFER probe whereas packets exceeding the MTU will be split into a list of packets and trigger the GST_PAD_PROBE_TYPE_BUFFER_LIST probe.

On the receiver, your probe on the sink pad of the RTP depayloader always receives a single packet from the UDP source, in this case GST_PAD_PROBE_TYPE_BUFFER is applicable.

Related