Description

I aim to generate a synthetic dataset using Blender. First, I would like to generate a scene with fisheye distortion using Blender, and then be able to undistort image/points with the OpenCV. I struggle with creating such function becouse of the following reasons.

It is possible to set fisheye polynomial distortion coefficients in Blender like this.

However, OpenCV fisheye distortion model looks different.

I discovered that it is not sufficient to match the coefficients with corresponding degree.

E.g. k0 has to be equal to zero, and k1 has to be equal to one in Blender notation. And then k2, k3, k4 in Blender notation does not correspond to the k1, k2, k3 in OpenCV notation.

What I tried

Camera Setup

I used a camera in (0, 0, 4) [meter] location (looking straight down), with 160 deg FOV. Sensor size 36mm. Render output resolution is (640, 480) and lens Fisheye polynomial coefficients are (0, 1, 0.3, 0, 0)

The image was imported as plane with dimensions (22.7, 17.8, 0) [meter]

Described setup resulted in the following image:

The original image looks like this:

Python Code

Then I tried to undistort the image with following code:

# distortion coefficients
D = np.array([0.3, 0, 0, 0], dtype=float)

# camera intrinsic coefficients obtained with the following function:
# https://blender.stackexchange.com/questions/38009/3x4-camera-matrix-from-blender-camera
K = np.array([[408.88888889, 0., 320.], [0., 460., 240.], [0., 0., 1.]])

visu = cv2.fisheye.undistortImage(img, K, D)

The code above resulted in a black screen.

However the code below resulted in almost unchanged image.

dim = img.shape[:2][::-1]
new_K = cv2.fisheye.estimateNewCameraMatrixForUndistortRectify(K, D, dim, np.eye(3), balance=0.5)
map1, map2 = cv2.fisheye.initUndistortRectifyMap(K, D, np.eye(3), new_K, dim, cv2.CV_16SC2)
visu = cv2.remap(img, map1, map2, interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT)