ARKit hide objects behind walls

Viewed 11953

How can I use the horizontal and vertical planes tracked by ARKit to hide objects behind walls/ behind real objects? Currently the 3D added objects can be seen through walls when you leave a room and/ or in front of objects that they should be behind. So is it possible to use the data ARKit gives me to provide a more natural AR experience without the objects appearing through walls?

5 Answers

For creating an occlusion material (also known as blackhole material or blocking material) you have to use the following instance properties: .colorBufferWriteMask, .readsFromDepthBuffer, .writesToDepthBuffer and .renderingOrder.

You can use them this way:

plane.geometry?.firstMaterial?.isDoubleSided = true
plane.geometry?.firstMaterial?.colorBufferWriteMask = .alpha  
plane.geometry?.firstMaterial?.writesToDepthBuffer = true
plane.geometry?.firstMaterial?.readsFromDepthBuffer = true
plane.renderingOrder = -100

...or this way:

func occlusion() -> SCNMaterial {

    let occlusionMaterial = SCNMaterial()
    occlusionMaterial.isDoubleSided = true
    occlusionMaterial.colorBufferWriteMask = []
    occlusionMaterial.readsFromDepthBuffer = true
    occlusionMaterial.writesToDepthBuffer = true

    return occlusionMaterial
}

plane.geometry?.firstMaterial = occlusion()
plane.renderingOrder = -100

In order to create an occlusion material it's really simple

    let boxGeometry = SCNBox(width: 0.1, height: 0.1, length: 0.1, chamferRadius: 0)

    // Define a occlusion material 
    let occlusionMaterial = SCNMaterial()
    occlusionMaterial.colorBufferWriteMask = []

    boxGeometry.materials = [occlusionMaterial]
    self.box = SCNNode(geometry: boxGeometry)
    // Set rendering order to present this box in front of the other models
    self.box.renderingOrder = -1

Great solution:

GitHub: arkit-occlusion

Worked for me.

But in my case i wanted to set the walls by code. So if you don't want to set the Walls by user -> use the plane detection to detect walls and set the walls by code.

Or in a range of 4 meters the iphone depht sensor works and you can detect obstacles with ARHitTest.

ARKit 4 and LiDAR scanner

You can hide any object behind a virtual invisible wall that replicates real wall geometry. iPhone 12 Pro and iPad Pro 4th Gen equipped with a LiDAR (Light Detection and Ranging) scanner help us reconstruct a 3d topological map of surrounding environment. LiDAR scanner greatly improves a quality of Z channel that allows occlude or remove humans from AR scene.

Also LiDAR improves such feature as Object Occlusion, Motion Tracking and Raycasting. With LiDAR scanner you can reconstruct a scene even in a unlit environment or in a room having white walls with no features at all. 3d reconstruction of surrounding environment has become possible in ARKit 4.0 thanks to sceneReconstruction instance property. Having a reconstructed mesh of your walls it's now super easy to hide any object behind real walls.

To activate a sceneReconstruction instance property in ARKit 4.0 use the following code:

@IBOutlet var arView: ARView!

arView.automaticallyConfigureSession = false

guard ARWorldTrackingConfiguration.supportsSceneReconstruction(.mesh)
else { return }

let config = ARWorldTrackingConfiguration()
config.sceneReconstruction = .mesh

arView.debugOptions.insert([.showSceneUnderstanding])
arView.environment.sceneUnderstanding.options.insert([.occlusion])
arView.session.run(config)


Also if you're using SceneKit try the following approach:

@IBOutlet var sceneView: ARSCNView!

func renderer(_ renderer: SCNSceneRenderer, 
          nodeFor anchor: ARAnchor) -> SCNNode? {

    guard let meshAnchor = anchor as? ARMeshAnchor 
    else { return nil }

    let geometry = SCNGeometry(arGeometry: meshAnchor.geometry)

    geometry.firstMaterial?.diffuse.contents = 
                            colorizer.assignColor(to: meshAnchor.identifier)
ā€
    let node = SCNNode()
    node.name = "Node_\(meshAnchor.identifier)"
    node.geometry = geometry
    return node
}

func renderer(_ renderer: SCNSceneRenderer,
          didUpdate node: SCNNode,
              for anchor: ARAnchor) {

    guard let meshAnchor = anchor as? ARMeshAnchor 
    else { return }

    let newGeometry = SCNGeometry(arGeometry: meshAnchor.geometry)

    newGeometry.firstMaterial?.diffuse.contents = 
                               colorizer.assignColor(to: meshAnchor.identifier)

    node.geometry = newGeometry
}

And here are SCNGeometry and SCNGeometrySource extensions:

extension SCNGeometry {
    convenience init(arGeometry: ARMeshGeometry) {
        let verticesSource = SCNGeometrySource(arGeometry.vertices, 
                                               semantic: .vertex)
        let normalsSource = SCNGeometrySource(arGeometry.normals, 
                                               semantic: .normal)
        let faces = SCNGeometryElement(arGeometry.faces)
        self.init(sources: [verticesSource, normalsSource], elements: [faces])
    }
}

extension SCNGeometrySource {
    convenience init(_ source: ARGeometrySource, semantic: Semantic) {
        self.init(buffer: source.buffer, vertexFormat: source.format,
                                             semantic: semantic,
                                          vertexCount: source.count,
                                           dataOffset: source.offset,
                                           dataStride: source.stride)
    }
}

...and SCNGeometryElement and SCNGeometryPrimitiveType extensions:

extension SCNGeometryElement {
    convenience init(_ source: ARGeometryElement) {
        let pointer = source.buffer.contents()
        let byteCount = source.count * 
                        source.indexCountPerPrimitive * 
                        source.bytesPerIndex
        let data = Data(bytesNoCopy: pointer, 
                              count: byteCount, 
                        deallocator: .none)
        self.init(data: data, primitiveType: .of(source.primitiveType),
                             primitiveCount: source.count,
                              bytesPerIndex: source.bytesPerIndex)
    }
}

extension SCNGeometryPrimitiveType {
    static func of(type: ARGeometryPrimitiveType) -> SCNGeometryPrimitiveType {
        switch type {
            case .line: return .line
            case .triangle: return .triangles
        }
    }
}
Related