Depth data from ARKit

user_1040898 0
Viewed 3815

Is it possible to access the AVDepthData from ARKit? The ARFrame contains the camera's image, but not its depth information.

I tried to create a separate AVCaptureSession to receive the AVDepthData, but I am unable to run the AVCaptureSession at the same time as ARKit. Either the ARSCNView updates or the AVCaptureDepthDataOutputDelegate gets called, but never both.

ios11 arkit avcapturesession
4 Answers

As was answered in this thread and mentioned in this video, no, ARKit does not provide you with AVDepthData when you are in world tracking mode. The only time when you are provided with AVDepthData is when you are in face tracking mode using an iPhone X.

user_28551730

This takes a long time (about 15 seconds), but I built a depth capture function using repeated calls to hitTest. This is probably horrendously non-optimal, but I couldn't figure out the best way to draw an image, and this is the first way I found that I really understood well enough to implement:

let height = Int(arView.frame.height)
let width = Int(arView.frame.width)


UIGraphicsBeginImageContextWithOptions(CGSize(width: width, height: height), true, 0)
while y < height {
  var x = 0
  while x < width {
    let location = CGPoint(x: x, y: y)
    let results = arView.hitTest(location, types: [.featurePoint, .existingPlane, .estimatedVerticalPlane, .estimatedHorizontalPlane, .existingPlaneUsingGeometry, .existingPlaneUsingExtent])
    let alpha = results.isEmpty ? 1.0 : (1.0 / CGFloat(results.count))
    for result in results {
      let value = 1.0 / (result.distance + 1.0)
      switch result.type {
        case .featurePoint:
          UIColor(red: value, green: 0, blue: 0, alpha: alpha).setFill()
        case .existingPlane:
          UIColor(red: 0, green: 1, blue: 0, alpha: alpha).setFill()
        case .estimatedVerticalPlane:
          UIColor(red: 0, green: 0, blue: value, alpha: alpha).setFill()
        case .estimatedHorizontalPlane:
          UIColor(red: 0, green: 0, blue: value, alpha: alpha).setFill()
        case .existingPlaneUsingGeometry:
          UIColor(red: value, green: value, blue: value, alpha: alpha).setFill()
        case .existingPlaneUsingExtent:
          UIColor(red: value, green: value, blue: value, alpha: alpha).setFill()
        default:
          UIColor.black.setFill()
      }
      UIRectFill(CGRect(x: x, y: y, width: 1, height: 1))
    }

    x += 1
  }

  y += 1

}

let image = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()

imgView.image = image

You can tweak the colors if you want. Anyway, I thought this gave me a pretty good glimpse of what ARKit thought about the world.

user_13269850

arkit also use AVCaptureSession, so to nowadays it's impossible to use arscnview while AVCaptureSession is going on, because it will call delegate - (void)sessionWasInterrupted:(ARSession *)session; Only method is use replay kit to record arkit screen.

user_22139990

In iOS 13 you can use frameSemantics 

let configuration = ARWorldTrackingConfiguration()
configuration.frameSemantics = .personSegmentationWithDepth

Then, in the ARSessionDelegate callback you can access the estimatedDepthData from the ARFrame

func session(_ session: ARSession, didUpdate frame: ARFrame) {
     let estimatedDepthData = frame.estimatedDepthData
     ....
}

You can also check

https://developer.apple.com/documentation/arkit/arframe/3152989-estimateddepthdata

https://developer.apple.com/documentation/arkit/arframe/2984226-segmentationbuffer

user_20596490
Related