iOS, How to do face tracking using the rear camera?

Question

I'm planning to use ARKit's camera feed as input into Apple's Vision API so I can recognize people's faces in screen-space, no depth information required. To simplify the process, I'm attempting to modify Apple's face tracking over frames example here: Tracking the User’s Face in Real Time

I thought that I could simply change the function here:

 fileprivate func configureFrontCamera(for captureSession: AVCaptureSession) throws -> (device: AVCaptureDevice, resolution: CGSize) {
        let deviceDiscoverySession = AVCaptureDevice.DiscoverySession(deviceTypes: [.builtInWideAngleCamera], mediaType: .video, position: .front)

        if let device = deviceDiscoverySession.devices.first {
            if let deviceInput = try? AVCaptureDeviceInput(device: device) {
                if captureSession.canAddInput(deviceInput) {
                    captureSession.addInput(deviceInput)
                }

                if let highestResolution = self.highestResolution420Format(for: device) {
                    try device.lockForConfiguration()
                    device.activeFormat = highestResolution.format
                    device.unlockForConfiguration()

                    return (device, highestResolution.resolution)
                }
            }
        }

        throw NSError(domain: "ViewController", code: 1, userInfo: nil)
    }

In the first line of the function, one of the arguments is .front for front-facing camera. I changed this to .back. This successfully gives me the rear-facing camera. However, the recognition region seems a little bit choppy, and as soon as it fixates on a face in the image, Xcode reports the error:

VisionFaceTrack[877:54517] [ServicesDaemonManager] interruptionHandler is called. -[FontServicesDaemonManager connection]_block_invoke
Message from debugger: Terminated due to memory issue

In other words, the program crashes when a face is recognized, it seems. Clearly there is more to this than simply changing the constant used. Perhaps there is a buffer somewhere with the wrong size, or a wrong resolution. May I have help figuring out what may be wrong here?

A better solution would also include information about how to achieve this with arkit's camera feed, but I'm pretty sure it's the same idea with the CVPixelBuffer.

How would I adapt this example to use the rear camera?

EDIT: I think the issue is that my device has too little memory to support the algorithm using the back camera, as the back camera has a higher resolution.

However, even on another higher performance device, the tracking quality is pretty bad. --yet the vision algorithm only needs raw images, doesn't it? In that case, shouldn't this work? I can't find any examples online of using the back camera for face tracking.

user_878695 · Answer

Here's how I adapted the sample to make it work on my iPad Pro.

1) Download the sample project from here: Tracking the User’s Face in Real Time.

2) Change the function which loads the front facing camera to use back facing. Rename it to configureBackCamera and call this method setupAVCaptureSession:

fileprivate func configureBackCamera(for captureSession: AVCaptureSession) throws -> (device: AVCaptureDevice, resolution: CGSize) {
    let deviceDiscoverySession = AVCaptureDevice.DiscoverySession(deviceTypes: [.builtInWideAngleCamera], mediaType: .video, position: .back)

    if let device = deviceDiscoverySession.devices.first {
        if let deviceInput = try? AVCaptureDeviceInput(device: device) {
            if captureSession.canAddInput(deviceInput) {
                captureSession.addInput(deviceInput)
            }

            if let highestResolution = self.highestResolution420Format(for: device) {
                try device.lockForConfiguration()
                device.activeFormat = highestResolution.format
                device.unlockForConfiguration()

                return (device, highestResolution.resolution)
            }
        }
    }

    throw NSError(domain: "ViewController", code: 1, userInfo: nil)
}

3) Change the implementation of the method highestResolution420Format. The problem is, now that the back-facing camera is used, you have access to formats with much higher resolution than with front-facing camera, which can impact the performance of the tracking. You need to adapt to your use case, but here's an example of limiting the resolution to 1080p.

fileprivate func highestResolution420Format(for device: AVCaptureDevice) -> (format: AVCaptureDevice.Format, resolution: CGSize)? {
    var highestResolutionFormat: AVCaptureDevice.Format? = nil
    var highestResolutionDimensions = CMVideoDimensions(width: 0, height: 0)

    for format in device.formats {
        let deviceFormat = format as AVCaptureDevice.Format

        let deviceFormatDescription = deviceFormat.formatDescription
        if CMFormatDescriptionGetMediaSubType(deviceFormatDescription) == kCVPixelFormatType_420YpCbCr8BiPlanarFullRange {
            let candidateDimensions = CMVideoFormatDescriptionGetDimensions(deviceFormatDescription)
            if (candidateDimensions.height > 1080) {
                continue
            }
            if (highestResolutionFormat == nil) || (candidateDimensions.width > highestResolutionDimensions.width) {
                highestResolutionFormat = deviceFormat
                highestResolutionDimensions = candidateDimensions
            }
        }
    }

    if highestResolutionFormat != nil {
        let resolution = CGSize(width: CGFloat(highestResolutionDimensions.width), height: CGFloat(highestResolutionDimensions.height))
        return (highestResolutionFormat!, resolution)
    }

    return nil
}

4) Now the tracking will work, but the face positions will not be correct. The reason is that UI presentation is wrong, because original sample was designed for front facing cameras with mirrored display, while the back facing camera doesn't need mirroring.

In order to tweak for this, simply change updateLayerGeometry() method. Specifically, you need to change this:

// Scale and mirror the image to ensure upright presentation.
let affineTransform = CGAffineTransform(rotationAngle: radiansForDegrees(rotation))
    .scaledBy(x: scaleX, y: -scaleY)
overlayLayer.setAffineTransform(affineTransform)

into this:

// Scale the image to ensure upright presentation.
let affineTransform = CGAffineTransform(rotationAngle: radiansForDegrees(rotation))
    .scaledBy(x: -scaleX, y: -scaleY)
overlayLayer.setAffineTransform(affineTransform)

After this, the tracking should work and the results should be correct.

iOS, How to do face tracking using the rear camera?

1 Answers