Homography from football (soccer) field lines

user_9435355 0
Viewed 3438

I am working on using video from a football (soccer) match and try to map the frames to a top view of the pitch using homography. I have started to get find all the white lines from the frames using both Hough lines as well as using the line segment detector, where the line segment detector seems to work slightly better. See my code and examples below:

import cv2
import numpy as np

cv2.imread("frame-27.jpg")
hsv = cv2.cvtColor(frame, cv2.COLOR_RGB2HSV)
mask_green = cv2.inRange(hsv, (36, 25, 25), (86, 255, 255)) # green mask to select only the field
frame_masked = cv2.bitwise_and(frame, frame, mask=mask_green)

gray = cv2.cvtColor(frame_masked, cv2.COLOR_RGB2GRAY)
_, thresh = cv2.threshold(gray, 127, 255, cv2.THRESH_BINARY)

canny = cv2.Canny(gray, 50, 150, apertureSize=3)
# Hough line detection
lines = cv2.HoughLinesP(canny, 1, np.pi / 180, 50, None, 50, 20)
# Line segment detection
lines_lsd = lsd.detect(canny)[0]

This uses this input frame

enter image description here

and returns this image for the Hough lines

enter image description here

and this image for the line segment detection.

enter image description here

My question is twofold: (1) any ideas on how I can further refine the line detection (i.e. decrease false positives such as lines around players and outside of the field) and (2) a good way to use the detection lines to create a homography so I can map the frame to a higher overview of the field (like this). Any help is greatly appreciated!

opencv3.0 opencv python
1 Answers

Pencils of lines

Cluster line segments in two pencils of lines using RANSAC. A pencil of lines is a set of lines that intersect at a common point. With homogeneous coordinates, the intersection point may potentially be at infinity (e.g. if all the lines are parallel).

You can find two random line segments, compute their intersection, and then consider all the lines that go somewhat close to that intersection point (within some threshold). Repeat this until you find the two pencils with the greatest amount of line segments. We can then assume that these pencils correspond to the vanishing points.

line segments

Here, the blue and red segments correspond to two pencils of lines. The green segments are outliers. As you can see the RANSAC algorithm is extremely good at rejecting outliers.

Rectifying

I am not aware of built-in OpenCV rectification for line segments specifically; the existing functions are designed to work with point correspondences.

Run an optimization to recover the homography to deal with orientation. Generally the homography H is of the form H = KRK^-1 where K is the intrinsic matrix and R is the rotation matrix.

For example, you can run a nonlinear least squares optimization on the manifold of the Lie group SO(3) to recover the R matrix. For example you can use LocalParameterization in Ceres solver. But it is pretty simple to implement this yourself in Python too. If the focal length is unknown, you'll have to add that as an optimization parameter as well.

Instead of nonlinear least squares there may be other methods. Some methods estimate the homography directly, but may not preserve the correct aspect ratio.

warped using homography

You can preview the homography by calling opencv's WarpPerspective function.

Estimate translation and scale

This will involve knowing something about the geometry of the football field. You can detect some salient features unique to the football field and estimate the scale. For example you can detect the circular arcs using the circle Hough transform.

football field superimposed

user_10125420
Related