GstSharp raw video pausing after gst_pad_push() = not-linked

Viewed 24

I've been working on a small Microsoft Teams extension that takes an incoming Audio/Video feed from each participant and records the data into an mp4 file for each participant. In all the MS Teams documentation they focus on getting access to the raw data itself and gloss over any kind of persistence of it into a usable video container. The video comes in as an NV12 encoded byte array and the audio is wav format. I've been trying to use Gstreamer to take the raw video data, push it to a videoconvert then on to a filesink to try and save the data as an h264 encoded mp4. I've pulled together a mix and match of code examples and think I'm close but missing something in the process. My pipeline creates successfully but when pushing buffers into the appsrc I get the following in my console:

pausing after gst_pad_push() = not-linked
error: Internal data stream error.
error: streaming stopped, reason not-linked (-1)

The code I use to create my pipeline, pads and sinks etc is as follows:

private Gst.App.AppSrc CreatePipeline(Guid identifier, string identity, int width, int height, string directory)
        {
            Directory.CreateDirectory($"{directory}\\temporary\\{identifier}");
            var pipeline = new Gst.Pipeline($"pipeline_{identity}");
            var VideoQueue = Gst.ElementFactory.Make("queue", $"video_queue_{identity}");
            var VideoConvert = Gst.ElementFactory.Make("videoconvert", $"video_convert_{identity}");
            var VideoSink = Gst.ElementFactory.Make("autovideosink", $"video_sink_{identity}");
            var AppQueue = Gst.ElementFactory.Make("queue", $"app_queue_{identity}");
            var AppSink = new Gst.App.AppSink($"app_sink_{identity}");
            var AppSource = new Gst.App.AppSrc($"app_src_{identity}");
            var Tee = Gst.ElementFactory.Make("tee", $"tee_{identity}");
            var FileSink = Gst.ElementFactory.Make("filesink", $"file_sink_{identity}");
            AppSource.Caps = Gst.Caps.FromString($"video/x-raw,format=NV12,width={width},height={height},framerate={this.fixedFps}");
            AppSource.IsLive = true;
            AppSink.EmitSignals = true;
            AppSink.Caps = Gst.Caps.FromString($"video/x-raw,format=NV12,width={width},height={height},framerate={this.fixedFps}");
            AppSink.NewSample += NewSample;
            Console.WriteLine("Setting Filesink location");
            FileSink["location"] = $"{directory}\\temporary\\{identifier}\\{identity}.mp4";

            pipeline.Add(AppSource, Tee, VideoQueue, VideoConvert, VideoSink, AppQueue, AppSink, FileSink);

            var teeVideoPad = Tee.GetRequestPad("src_%u");
            var queueVideoPad = VideoQueue.GetStaticPad("sink");
            var teeAppPad = Tee.GetRequestPad("src_%u");
            var queueAppPad = AppQueue.GetStaticPad("sink");

            if ((teeVideoPad.Link(queueVideoPad) != Gst.PadLinkReturn.Ok) ||
                (teeAppPad.Link(queueAppPad) != Gst.PadLinkReturn.Ok))
            {
                Console.WriteLine("Tee could not be linked.");
                throw new Exception("Tee could not be linked.");
            }

            AppSource.PadAdded += new Gst.PadAddedHandler(this.OnVideoPadAdded);
            var bus = pipeline.Bus;
            bus.AddSignalWatch();
            bus.Connect("message::error", HandleError);
            pipeline.SetState(Gst.State.Playing);
            return AppSource;
        }

And this gets called every time a new participant joins the call. Then each time a new video frame is sent to the call from a participant the following code is executed:

private bool NV12toMP4(byte[] array, string identity, int width, int height, long timestamp, int length)
        {

            var buffer = new Gst.Buffer(null, (ulong)length, Gst.AllocationParams.Zero)
            {
                Pts = (ulong)timestamp,
                Dts = (ulong)timestamp
            };
            buffer.Fill(0, array);
            var ret = this.participantSources[identity].PushBuffer(buffer);
            buffer.Dispose();
            if (ret != Gst.FlowReturn.Ok)
            {
                return false;
            }
            return true;
        }

I was expecting my PadAdded method to get called on the AppSrc when it first detects the type of input but this never gets triggered so I'm not sure where to look next on this.

0 Answers
Related