I'm looking for some direction on how to properly implement a Metal render loop. My render loop is being fed a stream of video frames from an AVPlayer.
Here's my current implementation:
- A
CVDisplayLinkqueries theAVPlayerItemVideoOutputof the player at 60hz. If a new frame is there, it'sCVPixelBufferRefis captured/saved* as a property of theMTKViewto which it will be rendered. (At this point, the previously captured video frame is released). - My
MTKViewis set up withisPausedandenableSetNeedsDisplaytoNO. In other words, the internal timer of theMTKViewis tasked with periodically calling itsdrawRectmethod. - In
drawRectI first convert the newly-arrived*CVPixelBufferto aMTLTexture, and then a bunch of rendering stages occur. - Finally, I call
presentDrawableat the end of thedrawRectmethod.
*Note: The mutex access to the CVPixelBufferRef is controlled by a pairing of dispatch_semaphore_wait and dispatch_semaphore_signal.
Is this a correct means of doing things? It seems fairly performant, though some frames are occasionally dropped. In terms of timing, an Xcode profiling of Metal tells me my MTLCommandBuffer is typically taking under 3ms to run.
Having said that, I see some alternate possibilities:
- ditch the
CVDisplayLinkimplementation and grab the frames inside ofdrawRect - reverse the rendering process; display the previously captured frame & rendered
MTLTexturefirst in thedrawRectmethod and thencommitthat Metal command buffer and callpresentDrawablepromptly. After that point, capture the next video frame and run it through its rendering stages before the nextdrawRectcall (do so beforedrawRectexits?).
Another issue: I was under the impression that both the CVDisplayLink and drawRect methods were not running on the main thread with this configuration. I'm troubled by the fact that whenever I let up on one of the app's menus, there's a significant stutter in the delivery of video frames -- this is symptomatic of the main thread doing UI updates and is blocking. The same behaviour is observed when an on-screen NSCollectionView is reloaded and its contents are animated on the screen. This leads me to believe my assumption is incorrect. How can these MTKView render loops be made to avoid these issues? Wondering if the entire opening/config of the AVPlayer needs to be off the main thread as well.
UPDATE #1
I fixed the "Another issue" stuttering issues which would occur when mousing up on any NSMenu items. This was my solution:
- Configure each
MTKViewwithisPaused=YESandenableSetNeedsDisplay=NO. This means that explicitdrawcalls are needed for a view to render its content. - From within the
CVDisplayLinkcallback, issue thedrawcall to theMTKViewon adispatch_global_queue, thusly:
__block MTKView *aView = ....;
dispatch_queue_t aQueue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
dispatch_async(aQueue, ^{ [aView draw]; });