ARSCNView snapshot() causes latency

user_1512804 0
Viewed 277

I'm taking a snapshot of every frame, applying a filter, and updating the background contents of the ARSCNView with the filtered image. Everything is working fine, but there is a lot of latency with all the UI elements on the screen. No latency on the ARSCNView.

 func session(_ session: ARSession, didUpdate frame: ARFrame) {

  guard let image = CIImage(image: sceneView.snapshot()) else { return }

  // I'm setting a filter to each image here. Which has no effect on the latency.

   sceneView.scene.background.contents = context.createCGImage(image, from: image.extent)
 }

I know I can use frame.capturedImage, which makes latency go away. However, I also place AR objects on the screen which frame.capturedImage ignores for some reason, and sceneView.scene.background.contents cannot be reset to its original source. So, I cannot turn off the image filter. That's why I need to take a snapshot.

Is there anything I can do that will reduce latency on the UI elements? I have a few UIScrollViews on the screen that have tremendous lag.

arscnview latency swift ios
1 Answers

I'm also in the middle of looking for a way to do this with no lag, but I was able to at least reduce the lag by rendering the view into an image manually:

extension ARSCNView {
    /// Performs screen snapshot manually, seems faster than built in snapshot() function, but still somewhat noticeable
    var snapshot: UIImage? {
       let renderer = UIGraphicsImageRenderer(size: self.bounds.size)
        let image = renderer.image(actions: { context in
            self.drawHierarchy(in: self.bounds, afterScreenUpdates: true)
        })
        return image
    }
}

It's frustrating that this is faster than the built-in snapshot function, but it seems to be, and also still captures all the SceneKit graphics in the snapshot. (Doing this every frame will still be expensive though, FYI, and the only real solution for that would likely be a custom Metal shader.)

I'm also trying to work with ARSCNView.snapshotView(afterScreenUpdates: Bool) because that seems to have essentially no lag for my purposes, but whenever I try to turn the resulting View into a UIImage, it's totally blank. Either way, the above method cut the lag in about half for me, so you might have some luck with that.

user_95050290
Related