I am trying to do 3D face reconstruction from two images using Python and OpecCV. I managed to perform camera calibration using intructions from here link, and then applying cv2.stereoCalibrate() and cv2.stereoRectify(). Afterwards, the cv2.StereoSGBM was used to compute the disparity map, and finally cv2.reprojectImageTo3D for generating the point cloud. The latter was displayed using open3d library. However, the result of StereoSGBM.compute() is very noisy, and the resulting point cloud looks nothing like it should.
Things I tried:
- normalizing the disparity map: The algorithm outputs a 2D array with values between -16 and 240, so I adjusted it to be between 0 and 255, and 0.0 and 1.0 respectively. The first option went better
- dividing the values by 16 (as I've seen in the C++ implementation)
- adjusting the paramaters of the StereoSGBM. I even built a GUI to tune the parameters, however the results were still not satisfactory
- using StereoBM, however there was not too much difference
- using different test images
- using graysale images
- normalizing them in range 0.0 -> 1.0
- cropping the undistorted images
- reimplementing
cv2.reprojectImageTo3D, with the same results - manually removing the background in the disparity map
- working with uint8 and float32. The reprojection worked way better using the floats
Resulting images:
initial:
after calibration:
disparity map (params on the right):
point cloud (looks human looking from a very specific angle):
What can I do to get usable results? I don't even know how the images should look like. Does my calibration seem off? I have a feeling the images are really bad (as the perspective distortion is pretty big), but I have to work with these unfortunately
Some code stubs:
_, self.camera_matrix, self.distortion, _, _ = \
cv2.calibrateCamera(self.object_points, self.image_points, self.image_size, None, None)
error, _, _, _, _, self.rotation, self.translation, _, _ = \
cv2.stereoCalibrate(self.camera_left.object_points,
self.camera_left.image_points,
self.camera_right.image_points,
self.camera_left.camera_matrix,
self.camera_left.distortion,
self.camera_right.camera_matrix,
self.camera_right.distortion,
self.camera_left.image_size,
flags=cv2.CALIB_FIX_INTRINSIC)
self.rotation_left, self.rotation_right, self.perspective_left, self.perspective_right, self.Q, self.roi_left, self.roi_right = \
cv2.stereoRectify(self.camera_left.camera_matrix,
self.camera_left.distortion,
self.camera_right.camera_matrix,
self.camera_right.distortion,
self.camera_left.image_size,
self.rotation,
self.translation,
flags=cv2.CALIB_ZERO_DISPARITY)
self.rotation_left, self.rotation_right, self.perspective_left, self.perspective_right, self.Q, self.roi_left, self.roi_right = \
cv2.stereoRectify(self.camera_left.camera_matrix,
self.camera_left.distortion,
self.camera_right.camera_matrix,
self.camera_right.distortion,
self.camera_left.image_size,
self.rotation,
self.translation,
flags=cv2.CALIB_ZERO_DISPARITY)
self.block_matching = cv2.StereoSGBM().create() # params not mentioned here, but they are set
disparity = self.block_matching.compute(undistorted_left, undistorted_right)
disparity = cv2.convertScaleAbs(disparity, beta=16)
point_cloud = cv2.reprojectImageTo3D(disparity_image, Q)
point_cloud = point_cloud.reshape(-1, point_cloud.shape[-1])
point_cloud = point_cloud[~np.isinf(point_cloud).any(axis=1)]
pcl = open3d.geometry.PointCloud()
pcl.points = open3d.utility.Vector3dVector(point_cloud)
open3d.visualization.draw_geometries([pcl])