AVCaptureVideoDataOutputSampleBufferDelegate drop frames using CIFilters for video filtering

I have very strange case where AVCaptureVideoDataOutputSampleBufferDelegate drops frames if I use 13 different filter chains. Let me explain:

I have CameraController setup, nothing special, here is my delegate method:

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
        if !paused {

            if connection.output?.connection(with: .audio) == nil {
                //capture video

                // my try to avoid "Out of buffers error", no luck ;(
                lastCapturedBuffer = nil
                let err = CMSampleBufferCreateCopy(allocator: kCFAllocatorDefault, sampleBuffer: sampleBuffer, sampleBufferOut: &lastCapturedBuffer)
                if err == noErr {

                }

                connection.videoOrientation = .portrait

                // getting image
                let pixelBuffer = CMSampleBufferGetImageBuffer(lastCapturedBuffer!)
                // remove if any
                CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))

                // captured - is just ciimage property
                captured = CIImage(cvPixelBuffer: pixelBuffer!)
                //remove if any
                CVPixelBufferUnlockBaseAddress(pixelBuffer!,CVPixelBufferLockFlags(rawValue: 0))
                //CVPixelBufferUnlockBaseAddress(pixelBuffer!, .readOnly)

                // transform image to targer resolution
                let srcWidth = CGFloat(captured.extent.width)
                let srcHeight = CGFloat(captured.extent.height)

                let dstWidth: CGFloat = ConstantsManager.shared.k_video_width
                let dstHeight: CGFloat = ConstantsManager.shared.k_video_height

                let scaleX = dstWidth / srcWidth
                let scaleY = dstHeight / srcHeight

                var transform = CGAffineTransform.init(scaleX: scaleX, y: scaleY)
                captured = captured.transformed(by: transform).cropped(to: CGRect(x: 0, y: 0, width: dstWidth, height: dstHeight))
                // mirror for front camera
                if front {
                    var t = CGAffineTransform.init(scaleX: -1, y: 1)
                    t = t.translatedBy(x: -ConstantsManager.shared.k_video_width, y: 0)
                    captured = captured.transformed(by: t)
                }

                // video capture logic
                let writable = canWrite()

                if writable,
                    sessionAtSourceTime == nil {
                    sessionAtSourceTime = CMSampleBufferGetPresentationTimeStamp(lastCapturedBuffer!)
                    videoWriter.startSession(atSourceTime: sessionAtSourceTime!)
                }

                if writable, (videoWriterInput.isReadyForMoreMediaData) {
                    videoWriterInput.append(lastCapturedBuffer!)
                }

                // apply effect in realtime <- here is problem. If I comment next line, it will be fixed but effect will n't be applied
                captured = FilterManager.shared.applyFilterForCamera(inputImage: captured)

                // current frame in case user wants to save image as photo
                self.capturedPhoto = captured

                // sent frame to Camcoder view controller
                self.delegate?.didCapturedFrame(frame: captured)
            } else {
                // capture sound
                let writable = canWrite()
                if writable, (audioWriterInput.isReadyForMoreMediaData) {
                    //print("write audio buffer")
                    audioWriterInput?.append(lastCapturedBuffer!)
                }
            }
        } else {
            // paused
        }
    }

I also implemented didDrop delegate method, here is how I figure out why it drops frames:

func captureOutput(_ output: AVCaptureOutput, didDrop sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
        print("did drop")
        var mode: CMAttachmentMode = 0
        let reason = CMGetAttachment(sampleBuffer, key: kCMSampleBufferAttachmentKey_DroppedFrameReason, attachmentModeOut: &mode)
        print("reason \(String(describing: reason))") // Optional(OutOfBuffers)
    }

So I did it like a pro and just commented parts of code to find where is the problem. So, it here:

captured = FilterManager.shared.applyFilterForCamera(inputImage: captured)

FilterManager - is singleton, here is called func:

func applyFilterForCamera(inputImage: CIImage) -> CIImage {
        return currentVsFilter!.apply(sourceImage: inputImage)
    }

currentVsFilter is object of VSFilter type - here is example of one:

import Foundation
import AVKit

class TestFilter: CustomFilter {

    let _name = "Тестовый Фильтр"
    let _displayName = "Test Filter"

    var tempImage: CIImage?
    var final: CGImage?

    override func name() -> String {
        return _name
    }

    override func displayName() -> String {
        return _displayName
    }

    override init() {
        super.init()
        print("Test Filter init")

        // setup my custom kernel filter
        self.noise.type = GlitchFilter.GlitchType.allCases[2]
    }

    // this returns composition for playback using AVPlayer
    override func composition(asset: AVAsset) -> AVMutableVideoComposition {
        let composition = AVMutableVideoComposition(asset: asset, applyingCIFiltersWithHandler: { request in
            let inputImage = request.sourceImage.cropped(to: request.sourceImage.extent)
            DispatchQueue.global(qos: .userInitiated).async {
                let output = self.apply(sourceImage: inputImage, forComposition: true)
                request.finish(with: output, context: nil)
            }
        })
        let size = FilterManager.shared.cropRectForOrientation().size

        composition.renderSize = size
        return composition
    }

    // this returns actual filtered CIImage, used for both AVPlayer composition and realtime camera
    override func apply(sourceImage: CIImage, forComposition: Bool = false) -> CIImage {

        // rendered text
        tempImage = FilterManager.shared.textRenderedImage()

        // some filters chained one by one
        self.screenBlend?.setValue(tempImage, forKey: kCIInputImageKey)
        self.screenBlend?.setValue(sourceImage, forKey: kCIInputBackgroundImageKey)

        self.noise.inputImage = self.screenBlend?.outputImage
        self.noise.inputAmount = CGFloat.random(in: 1.0...3.0)

        // result
        tempImage = self.noise.outputImage

        // correct crop
        let rect = forComposition ? FilterManager.shared.cropRectForOrientation() : FilterManager.shared.cropRect
        final = self.context.createCGImage(tempImage!, from: rect!)

        return CIImage(cgImage: final!)
    }

}

And now, the most strange thing, I have 30 VSFilters and when I got to 13(switching one by one by UIButton) I got error "Out of Buffer", this one:

kCMSampleBufferDroppedFrameReason_OutOfBuffers

What I tested:

• I changed vsFilters order in filters array inside FilterManager singleton - same
• I tried switch from first to 12 one by one, then go back - works, but after I switched to 13tn(of 30th from 0) - bug

Looks like it can handle only 12 VSFIlter objects, like if it retains them somehow or maybe it's related to threading, I don't know.

This app made for iOs devices, tested on iPhone X iOs 13.3.1 This is video editor app to apply different effects to both live stream from camera and video files from camera roll

Maybe someone has experience with this?

Have a great day

Best, Victor

Edit 1. If I reinit cameraController(AVCaptureSession. input/output devices) it works but this is ugly option and it adds lag when switching filters