Efficiently updating ARSCNFaceGeometry from a set of blend shapes

Viewed 645

I am using ARSCNFaceGeometry and need to update the face model's blend shapes as part of my game loop. My current solution is to call ARSCNFaceGeometry.update with a new ARFaceGeometry:

class Face {
    let geometry: ARSCNFaceGeometry
    
    init(device: MTLDevice) {
        guard let geometry = ARSCNFaceGeometry(device: device, fillMesh: true) else {
            fatalError("Could not create ARSCNFaceGeometry")
        }
        self.geometry = geometry
    }
    
    func update(blendShapes: [ARFaceAnchor.BlendShapeLocation: NSNumber]) {
        let faceGeometry = ARFaceGeometry(blendShapes: blendShapes)!
        geometry.update(from: faceGeometry)
    }
}

However this is not suitable for realtime usage since the ARFaceGeometry line alone takes around 0.01 seconds (just for reference, at 60fps we have a total frame budget of 0.0166 seconds).

After a few hundred updates or so, we seem to run to some sort of bug that takes the entire game loop to 10-20fps. Here's a 6 second sample from the profiler that sees ARFaceGeometry taking 2.3 seconds!:

ARFaceGeometry taking 2.3 seconds in a 6 second sample

Is there a more efficient way to update an existing ARSCNFaceGeometry from a set of blend shapes?

With custom 3D models for example, I can just update the blend shape values on SCNNode.morpher. Is there an equivalent for ARSCNFaceGeometry?

2 Answers

Apple Developer Documentation says:

To update a SceneKit model of a face actively tracked in an AR session, call this method in your ARSCNViewDelegate object’s renderer(_:didUpdate:for:) callback, passing the geometry property from the ARFaceAnchor object that callback provides.

First code version

extension ViewController: ARSCNViewDelegate {

    func renderer(_ renderer: SCNSceneRenderer,
              nodeFor anchor: ARAnchor) -> SCNNode? {

        let faceGeo = ARSCNFaceGeometry(device: sceneView.device!)
        let node = SCNNode(geometry: faceGeo)
        node.geometry?.firstMaterial?.fillMode = .lines
        return node
    }

    func renderer(_ renderer: SCNSceneRenderer,
              didUpdate node: SCNNode,
                  for anchor: ARAnchor) {

        if let faceAnchor = anchor as? ARFaceAnchor,
           let faceGeo = node.geometry as? ARSCNFaceGeometry {

            if faceAnchor.lookAtPoint.x >=  0 {
                faceGeo.firstMaterial?.diffuse.contents = UIColor.red
            } else {
                faceGeo.firstMaterial?.diffuse.contents = UIColor.cyan
            }

            faceGeo.update(from: faceAnchor.geometry)
            self.facialExrpession(anchor: faceAnchor)

            DispatchQueue.main.async {
                self.label.text = self.textBoard
            }
        }
    }
}

...

extension ViewController {

    func facialExrpession(anchor: ARFaceAnchor) {

        self.textBoard = "PLEASE SMILE!"
        let smileLeft = anchor.blendShapes[.mouthSmileLeft]
        let smileRight = anchor.blendShapes[.mouthSmileRight]

        if ((smileLeft?.decimalValue ?? 0.0) + 
            (smileRight?.decimalValue ?? 0.0)) > 0.5 {
            self.textBoard += "You are smiling"
        }
    }
}


Second code version

import ARKit
import Metal

class BlendShapes: NSObject, ARSCNViewDelegate {

    typealias BlendShapeDict = [ARFaceAnchor.BlendShapeLocation: NSNumber]
    var collectBlendShapes: ((BlendShapeDict) -> Void)?

    let geometry = ARSCNFaceGeometry(device: MTLCreateSystemDefaultDevice()!, 
                                   fillMesh: true)
    
    func renderer(_ renderer: SCNSceneRenderer,
              didUpdate node: SCNNode,
                  for anchor: ARAnchor) {

        guard let facialAnchor = anchor as? ARFaceAnchor else { return }
        let blendShapes: (Any)? = collectBlendShapes?(facialAnchor.blendShapes)
        
        geometry!.update(from: blendShapes as! ARFaceGeometry)
    }
}

Alternatively, you can create, configure, and visualize face models independent of an AR session by creating face geometry objects using the ARFaceGeometry init(blendShapes:) initializer and passing them to this method.

I could not track down the ARFaceGeometry performance problems. To workaround this, I instead decided to build my own model from ARFaceGeometry.

First I generated a model file from ARFaceGeometry. This file includes the base geometry as well as the geometry when each individual blend shape is applied:

let LIST_OF_BLEND_SHAPES: [ARFaceAnchor.BlendShapeLocation] = [
    .eyeBlinkLeft,
    .eyeLookDownLeft,
    // ... fill in rest
]

func printFaceModelJson() {
    // Get the geometry without any blend shapes applied 
    let base = ARFaceGeometry(blendShapes: [:])!
    
    // First print out a single copy of the indices.
    // These are shared between the models
    let indexList = base.triangleIndices.map({ "\($0)" }).joined(separator: ",")
    print("indexList: [\(indexList)]")
    
    // Then print the starting geometry (i.e. no blend shapes applied)
    printFaceNodeJson(blendShape: nil)
    
    // And print the model with each blend shape applied
    for blend in LIST_OF_BLEND_SHAPES {
        printFaceNodeJson(blendShape: blend)
    }
}

func printFaceNodeJson(
    blendShape: ARFaceAnchor.BlendShapeLocation?
) {
    let geometry = ARFaceGeometry(blendShapes: blendShape != nil ? [blendShape!: 1.0] : [:])!
    
    let verticies = geometry.vertices.flatMap({ v in [v[0], v[1], v[2]] })
    let vertexList = verticies.map({ "\($0)" }).joined(separator: ",")
    print("{ \"blendShape\": \(blendShape != nil ?  "\"" + blendShape!.rawValue + "\"" : "null"), \"verticies\": [\(vertexList)] }")
}

I ran this code offline to generate the model file (quickly converting the output by hand to proper json). You could also use a proper 3D model file format, which would likely result in a smaller model file.

Then for my app, I reconstruct the model from the json model file:

class ARMaskFaceModel {
    
    let node: SCNNode
    
    init() {
        let data = loadMaskJsonDataFromFile() // implement this!
        
        let elements = [SCNGeometryElement(indices: data.indicies, primitiveType: .triangles)]
        
        // Create the base geometry
        let baseGeometryData = data.blends[0]
        let geometry = SCNGeometry(sources: [
            SCNGeometrySource(vertices: baseGeometryData.verticies)
        ], elements: elements)
        
        node = SCNNode(geometry: geometry)
        
        // Then load each of the blend shape geometries into a morpher
        let morpher = SCNMorpher()
        morpher.targets = data.blends.dropFirst().map({ x in
            SCNGeometry(sources: [
                SCNGeometrySource(vertices: x.verticies)
            ], elements: elements)
        })
        node.morpher = morpher
    }

    /// Apply blend shapes to the model
    func update(blendShapes: [ARFaceAnchor.BlendShapeLocation : NSNumber]) {
        var i = 0
        for blendShape in LIST_OF_BLEND_SHAPES {
            if i > node.morpher?.targets.count ?? 0 {
                return
            }
            node.morpher?.setWeight(CGFloat(truncating: blendShapes[blendShape] ?? 0.0), forTargetAt: i)
            i += 1
        }
    }
}

Not ideal, but it works ok and performs much better, even without any optimization. We're pretty consistently at 60fps now. Plus it works on older phones too! (although printFaceModelJson must be run on a phone that supports realtime face tracking)

Related