High CPU and energy use by iOS Vision framework face tracking

user_1204766 0
Viewed 512

I am prototyping an app which would use the iOS Vision framework to continuously crop tightly around the user's face through the front camera. I've found a great tutorial from the official Apple documentation: https://developer.apple.com/documentation/vision/tracking_the_user_s_face_in_real_time

My plan is to adapt this to crop the video instead of drawing a bounding box, and then build the rest of my app around that.

However, I've noticed that this example app uses a high amount of energy and CPU. The app will not be viable if this is unavoidable, as it will quickly drain the user's battery.

Is the Vision framework intended just to be used in short bursts? Is it worth diving into trying to optimize it?

computer-vision swift ios
1 Answers

I have modified this tutorial: https://www.raywenderlich.com/1163620-face-detection-tutorial-using-the-vision-framework-for-ios

To only use the VNDetectFaceLandmarksRequest method every 5th frame. This has brought CPU usage down to an acceptable level while still having acceptable performance on following the face. I will try to adapt the official tutorial as well.

  var frameCounter: UInt64 = 0
  func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    frameCounter += 1
    if frameCounter % 5 == 0 {
      // 1
      guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
        return
      }

      // 2
      let detectFaceRequest = VNDetectFaceLandmarksRequest(completionHandler: detectedFace)

      // 3
      do {
        try sequenceHandler.perform(
          [detectFaceRequest],
          on: imageBuffer,
          orientation: .leftMirrored)
      } catch {
        print(error.localizedDescription)
      }
    }
  }
user_12047660
Related