Move Camera Over Terrain Using Touch Input in Unity 3D

Pavel's answer helped me a lot, so wanted to share my solution with the community in case it helps others. My scenario is a 3D world with an orthographic camera. A top-down style RTS I am working on. I want pan and zoom to work like Google Maps, where the mouse always stays at the same spot on the map when you pan and zoom. This script achieves this for me, and hopefully is robust enough to work for others' needs. I haven't tested it a ton, but I commented the heck out of it for beginners to learn from.

using UnityEngine;

// I usually attach this to my main camera, but in theory you can attach it to any object in scene, since it uses Camera.main instead of "this".
public class CameraMovement : MonoBehaviour
{
    private Vector3 MouseDownPosition;

    void Update()
    {
        // If mouse wheel scrolled vertically, apply zoom...
        // TODO: Add pinch to zoom support (touch input)
        if (Input.mouseScrollDelta.y != 0)
        {
            // Save location of mouse prior to zoom
            var preZoomPosition = getWorldPoint(Input.mousePosition);

            // Apply zoom (might want to multiply Input.mouseScrollDelta.y by some speed factor if you want faster/slower zooming
            Camera.main.orthographicSize = Mathf.Clamp(Camera.main.orthographicSize + Input.mouseScrollDelta.y, 5, 80);

            // How much did mouse move when we zoomed?
            var delta = getWorldPoint(Input.mousePosition) - preZoomPosition;

            // Rotate camera to top-down (right angle = 90) before applying adjustment (otherwise we get "slide" in direction of camera angle).
            // TODO: If we allow camera to rotate on other axis we probably need to adjust that also.  At any rate, you want camera pointing "straight down" for this part to work.
            var rot = Camera.main.transform.localEulerAngles;
            Camera.main.transform.localEulerAngles = new Vector3(90, rot.y, rot.z);

            // Move the camera by the amount mouse moved, so that mouse is back in same position now.
            Camera.main.transform.Translate(delta.x, delta.z, 0);

            // Restore camera rotation
            Camera.main.transform.localEulerAngles = rot;
        }

        // When mouse is first pressed, just save location of mouse/finger.
        if (Input.GetMouseButtonDown(0))
        {
            MouseDownPosition = getWorldPoint(Input.mousePosition);
        }

        // While mouse button/finger is down...
        if (Input.GetMouseButton(0))
        {
            // Total distance finger/mouse has moved while button is down
            var delta = getWorldPoint(Input.mousePosition) - MouseDownPosition;

            // Adjust camera by distance moved, so mouse/finger stays at exact location (in world, since we are using getWorldPoint for everything).
            Camera.main.transform.Translate(delta.x, delta.z, 0);
        }
    }

    // This works by casting a ray.  For this to work well, this ray should always hit your "ground".  Setup ignore layers if you need to ignore other colliders.
    // Only tested this with a simple box collider as ground (just one flat ground).
    private Vector3 getWorldPoint(Vector2 screenPoint)
    {
        RaycastHit hit;
        Physics.Raycast(Camera.main.ScreenPointToRay(screenPoint), out hit);
        return hit.point;
    }
}

Based on the answer from Pavel, I simplified the script and removed the unlovely "jump" when touch with more then one finger and release the second finger:

private bool moreThenOneTouch = false;
private Vector3 worldStartPoint;

void Update() {

    Touch currentTouch;
    // only work with one touch
    if (Input.touchCount == 1 && !moreThenOneTouch) {
        currentTouch = Input.GetTouch(0);

        if (currentTouch.phase == TouchPhase.Began) {
            this.worldStartPoint = Camera.main.ScreenToWorldPoint(currentTouch.position);
        }

        if (currentTouch.phase == TouchPhase.Moved) {
            Vector3 worldDelta = Camera.main.ScreenToWorldPoint(currentTouch.position) - this.worldStartPoint;
            
            Camera.main.transform.Translate(
                -worldDelta.x,
                -worldDelta.y,
                0
            );
        }
    
    }

    if (Input.touchCount > 1) {
        moreThenOneTouch = true;
    } else {
        moreThenOneTouch = false;
        if(Input.touchCount == 1)
            this.worldStartPoint = Camera.main.ScreenToWorldPoint(Input.GetTouch(0).position);
    }
}