Multiple AVCaptureSession on macOS. What is the right approach?

user_6624755 0
Viewed 153

I want to preview and record more than one AVCaptureDevice on macOS using AVFoundation. I have a test app, where I create AVCaptureSession for every device input I want to preview and record. I add AVCaptureInput, AVCaptureMovieFileOutput and AVCaptureVideoPreviewLayer to every AVCaptureSession. I have a global object, 'MyAVSessionManager', it manages all sessions. All sessions share just one 'dispatch_queue_create("sr_capture_queue_video", DISPATCH_QUEUE_SERIAL);'. Everything works, but when I connect two devices (it doesn't matter if their frame rate is identical or not), GPU framerate doubles from 50fps to around 100fps (both inputs are connected to the different signals (but identical formats. I used 1080p50 for testing). My conclusion (I could be totally wrong) is that blit from the captured video buffers is not synchronized between the sessions, so every session writes its video buffer when it arrives. I didnt' find an option how to synchronize it (AVCaptureSession?), so videobuffers will be blit to my preview layers at once, in sync with display. I want to try to use one common AVCapture session for all devices, with multiple AVCaptureVideoPreviewLayers and AVCaptureMovieFileOutputs (connected to the appropriate inputs). I hope the blit from different video buffers could be synced in this case. If this fails, the only option I see is to have multiple AVCaptureVideoDataOutputs, use Metal to create texture from incoming CVImageBuffer and sync commandBuffer.commit() for all buffers. My questions: Which approach is the best? What is better: to create separate AVCaptureSession for every device (with shared queue or own queue?) or one, for all devices? What is the right way to display (and synchronize) multiple previews? Thanks.

avcapturesession avfoundation macos
1 Answers

Fix: add all connections manually. What I did wrong is this:

       let previewOutput = AVCapturePhotoOutput()
    
        // Add a previewOutput.
        guard session.canAddOutput(previewOutput) else {
            print("Could not add previewOutput to the session")
            setupResult = .configurationFailed
            session.commitConfiguration()
            return
        }
        
        session.addOutput(previewOutput)
        
        let previewLayer  = AVCaptureVideoPreviewLayer(sessionWithNoConnection: session)
        let previewLayerConnection = AVCaptureConnection(inputPort: devicePort, videoPreviewLayer: previewLayer)
        
        guard session.canAddConnection(previewLayerConnection) else
        {   print("Could not add preview layer connection to the session")
            setupResult = .configurationFailed
            session.commitConfiguration()
            return
        }
        
        session.addConnection(previewLayerConnection)

This is wrong: session.addOutput(previewOutput) It must be: session.addOutputWithNoConnections(previewOutput) and then create connection manually:

        let previewLayer  = AVCaptureVideoPreviewLayer(sessionWithNoConnection: session)
        let previewLayerConnection = AVCaptureConnection(inputPort: devicePort, videoPreviewLayer: previewLayer)
        
        guard session.canAddConnection(previewLayerConnection) else
        {   print("Could not add preview layer connection to the session")
            setupResult = .configurationFailed
            session.commitConfiguration()
            return
        }
        
        session.addConnection(previewLayerConnection)

I created this connection twice. When I checked the session, there was the correct number of connections (via print description). I don't know what happens with the connection I created using session.addOutput() (it wasn't listed in the session.connections), but the problem with the FPS, I noticed in gauges, disappeared

user_66247550
Related