How to get physics to work in collaborative sessions using RealityKit?

I am trying to create a collaborative AR App, that allows multiple users to add blocks to the scene, which should have physics and interact with each other. To do that I am using RealityKit.

I set up the collaborative session with MultipeerConnectivity and did get the connectivity to work. Both devices can see the blocks placed by the other device. Everything works fine until I add physics. With physics, everything starts to behave strangely. Blocks are jumping around and I will not have the same behavior on both systems.

I then tried to separate host and client, so that only the host places ModelEntities. So I now have a TapGestureRecognizer, that places ARAnchors

@objc func handleTap(recognizer: UITapGestureRecognizer){
    let cameraPosition = arView.cameraTransform.matrix
    let translate = float4x4(
        [1,0,0,0],
        [0,1,0,0],
        [0,0,1,0],
        [0,0,-0.4,1]
    )
    let blockPosition = cameraPosition * translate
    
    let arAnchor = ARAnchor(name: "block", transform: blockPosition)
    arView.session.add(anchor: arAnchor)
}

And I implemented the delegate method to let the host add a ModelEntity, once an ARAnchor was added:

func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
    for anchor in anchors {
        if anchor.name == "block" {
            
            // Check for host or not
            // If host: add block to the scene
            if isHost {
                let anchorEntity = AnchorEntity(anchor: anchor)
                
                let model = ModelEntity(
                    mesh: .generateBox(size: [0.3, 0.05, 0.2]),
                    materials: [SimpleMaterial(color: .cyan, isMetallic: false)]
                )
                model.generateCollisionShapes(recursive: true)
                model.physicsBody = PhysicsBodyComponent(
                    shapes: model.collision!.shapes,
                    mass: 1,
                    material: .generate(friction: 10, restitution: 0.1),
                    mode: .dynamic
                )
                
                anchorEntity.addChild(model)
                
                anchorEntity.synchronization?.ownershipTransferMode = .autoAccept
                arView.scene.addAnchor(anchorEntity)
            }
        }
    }
}

On the host device, everything works fine. But on the physics behave completely different. Blocks are jumping around. I can see, that when I add a new block it seems to try to synchronize both worlds again, and for a short moment on the client system the blocks are at the same place as on the host system, but then the blocks jump out of each other again.

Has anybody faced similar problems and found a solution to this? Or am I doing something wrong?

Thanks in advance!