Unity Orthographic Camera Panning

(I'm relatively new to Unity) I'm making a scene where I want the camera to move in the opposite direction of the mouse movement when I hold down the right mouse button. The scene is isotropic, which is why the camera is orthographic.

[SerializeField] private Camera cam;
private Vector3 dragOrigin = Vector3.zero;
public float speed = 10;
    
void Update(){
    if(Input.GetMouseButtonDown(1)){
        dragOrigin = cam.ScreenToWorldPoint(Input.mousePosition);
    }
    if(Input.GetMouseButton(1)){
        Vector3 difference = cam.ScreenToWorldPoint(Input.mousePosition) - cam.transform.position;
        Vector3 targetPosition = dragOrigin - difference;
        //targetPosition.y = cam.transform.position.y;
            
        cam.transform.position = Vector3.Lerp(cam.transform.position, targetPosition, speed * Time.deltaTime);
    }
}

This works, but there is the problem that the y-value of the camera changes as well. It should always stay at the same height. If I use targetPosition.y = cam.transform.position.y; the camera stays on the same y-position, but it moves slower when I move it up and down.

How can I rewrite the code so that only the x and z values change and that the camera moves with the same speed in all directions?

Thanks in advance.