Metal Render Loop optimization

Viewed 727

I'm looking for some direction on how to properly implement a Metal render loop. My render loop is being fed a stream of video frames from an AVPlayer.

Here's my current implementation:

  1. A CVDisplayLink queries the AVPlayerItemVideoOutput of the player at 60hz. If a new frame is there, it's CVPixelBufferRef is captured/saved* as a property of the MTKView to which it will be rendered. (At this point, the previously captured video frame is released).
  2. My MTKView is set up with isPaused and enableSetNeedsDisplay to NO. In other words, the internal timer of the MTKView is tasked with periodically calling its drawRect method.
  3. In drawRect I first convert the newly-arrived* CVPixelBuffer to a MTLTexture, and then a bunch of rendering stages occur.
  4. Finally, I call presentDrawable at the end of the drawRect method.

*Note: The mutex access to the CVPixelBufferRef is controlled by a pairing of dispatch_semaphore_wait and dispatch_semaphore_signal.

Is this a correct means of doing things? It seems fairly performant, though some frames are occasionally dropped. In terms of timing, an Xcode profiling of Metal tells me my MTLCommandBuffer is typically taking under 3ms to run.

Having said that, I see some alternate possibilities:

  • ditch the CVDisplayLink implementation and grab the frames inside of drawRect
  • reverse the rendering process; display the previously captured frame & rendered MTLTexture first in the drawRect method and then commit that Metal command buffer and call presentDrawable promptly. After that point, capture the next video frame and run it through its rendering stages before the next drawRect call (do so before drawRect exits?).

Another issue: I was under the impression that both the CVDisplayLink and drawRect methods were not running on the main thread with this configuration. I'm troubled by the fact that whenever I let up on one of the app's menus, there's a significant stutter in the delivery of video frames -- this is symptomatic of the main thread doing UI updates and is blocking. The same behaviour is observed when an on-screen NSCollectionView is reloaded and its contents are animated on the screen. This leads me to believe my assumption is incorrect. How can these MTKView render loops be made to avoid these issues? Wondering if the entire opening/config of the AVPlayer needs to be off the main thread as well.

UPDATE #1

I fixed the "Another issue" stuttering issues which would occur when mousing up on any NSMenu items. This was my solution:

  1. Configure each MTKView with isPaused=YES and enableSetNeedsDisplay=NO. This means that explicit draw calls are needed for a view to render its content.
  2. From within the CVDisplayLink callback, issue the draw call to the MTKView on a dispatch_global_queue, thusly:

__block MTKView *aView = ....;
dispatch_queue_t aQueue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
dispatch_async(aQueue, ^{ [aView draw]; });
0 Answers
Related