Create an Undistorted Top-Down View of Camera Image

user_19623095 0
Viewed 88

I have a fixed camera mounted on a wall viewing a rectangular lawn at an angle. My goal is to obtain an undistorted, top-down view of the lawn.

I have an image from the camera as a python numpy array which looks like this:

raw camera image

I use an inverse matrix with skimage.transform.warp to correct the image to a top down view:

top down distorted

This works perfectly, however the camera lens introduces barrel distortion.

Seperately, I can correct the distortion with a generated lookup table using skimage.transform.warp_coords and passing a simple undistort callable function based on the algorithm described here. The image is then generated using scipy.ndimage.map_coordinates:

undistorted camera view

These 2 processes work individually, but how do I combine them to create an undistorted top-down view, without creating an intermediate image?

I could run each point in the lookup table through the matrix to create a new table, but the table is massive and memory is tight (Raspberry Pi Zero).

I would like to define the undistortion as a matrix and just combine the 2 matrices, but as I understand it, the projective homography matrix is linear but undistortion is non-linear, so this can't be done. I can't use OpenCV due to resource constraints, and the calibration procedure involving multiple chessboard images is impractical. Currently, I calibrate by taking 4 lawn corner points and generate the matrix from them, which works well.

I would have anticipated that this is a common problem in Computer Vision but can't find any suitable solutions.

homography scipy computer-vision numpy python
2 Answers

The barrel distortion is nonlinear, but it is also smooth. This means it can be well approximated by a collection of piecewise linear approximations. So you do not need a large, per-pixel look-up table of un-distortion displacements. Rather, you can subsample it (or just scale it down), and use bilinear interpolation for in-between pixels.

user_14352400

I have found a solution that appears to work by creating seperate functions for undistort and transformation, then chaining them together.

The skimage source code here has the _apply_mat method for generating a mapping from a matrix. I based my unwarp function on that:

def unwarp(coords, matrix):
    coords = np.array(coords, copy=False, ndmin=2)
    x, y = np.transpose(coords)
    src = np.vstack((x, y, np.ones_like(x)))
    dst = src.T @ matrix.T

    # below, we will divide by the last dimension of the homogeneous
    # coordinate matrix. In order to avoid division by zero,
    # we replace exact zeros in this column with a very small number.
    dst[dst[:, 2] == 0, 2] = np.finfo(float).eps
    # rescale to homogeneous coordinates
    dst[:, :2] /= dst[:, 2:3]

    return dst[:, :2]

I created a similar function for undistorting based on Tanner Hellands algorithm:

def undistort(coords, cols, rows, correction_radius, zoom):
    half_width = cols / 2
    half_height = rows / 2
    new_x = coords[:, 0] - half_width
    new_y = coords[:, 1] - half_height
    distance = np.hypot(new_x, new_y)
    r = distance / correction_radius
    theta = np.ones_like(r)
    # only process non-zero values
    np.divide(np.arctan(r), r, out=theta, where=r!=0)
    source_x = half_width + theta * new_x * zoom
    source_y = half_height + theta * new_y * zoom
    result = np.column_stack([source_x, source_y])       
    return result

The only tricky bit here is the divide where we need to prevent division by zero.

Once we have each lookup table we can chain them together:

def undistort_unwarp(coords):
    undistorted = undistort(coords)
    both = unwarp(undistorted)
    return both

Note that these are the callable functions passed to skimage.transform.warp_coords:

mymap = tf.warp_coords(undistort_unwarp, shape=(rows, cols), dtype=np.int16)

The map can then be passed to the skimage.transform.warp function.

Francesco's answer was helpful, however I needed the full pixel resolution for the transformation, so I used it for the undistort as well, and looked to other ways to reduce the memory consumption.

Each map consumes

rows * cols * bytes-per-item * 2 (x and y)

bytes. The default datatype is float64, which requires 8 bytes-per-item, and the documentation suggests sane choices would be the default or float32 at 4 bytes-per-item. I was able to reduce this to 2 bytes-per-item using int16 with no visible ill effects, but I suspect the spline interpolation is not being used to the full (at all?).

The map is the same for each channel of a colour RGB image. However, when I called warp_coords with shape=(rows, cols, 3) I got 3 duplicate maps, so I created a function to handle colour images by processing each channel separately:

def warp_colour(img_arr, coord_map):
    if img_arr.ndim == 3:
        # colour
        rows, cols, _chans = img_arr.shape
        r_arr = tf.warp(img_arr[:, :, 0], inverse_map=coord_map, output_shape=(rows, cols))
        g_arr = tf.warp(img_arr[:, :, 1], inverse_map=coord_map, output_shape=(rows, cols))
        b_arr = tf.warp(img_arr[:, :, 2], inverse_map=coord_map, output_shape=(rows, cols))
        rgb_arr = np.dstack([r_arr, g_arr, b_arr])
    else:
        # grayscale
        rows, cols = img_arr.shape
        rgb_arr = tf.warp(img_arr, inverse_map=coord_map, output_shape=(rows, cols))
    return rgb_arr

One issue with skimage.transform.warp_coords is that it does not have the map_args dictionary parameter that skimage.transform.warp has. I had to call my unwarp and undistort functions through an intermediate function to add the parameters.

user_196230950
Related