OpenCV stereoCalibration of two cameras using ChArUco

Question

I want to calculate the relative transformation between two cameras ([R|t] matrix) using multiple frames of a charuco board. My idea was to obtain image-object point pairs from all the frames and then use a function which takes all of the detected point pairs and outputs relative transformation between cameras (e.g. stereoCalibrate).

What is the best approach to do that? I could not get stereoCalibrate to work, since it always throws assertion errors -> bugreport.

Current implementation (not working):

imagePointsA = []
imagePointsB = []
objectPoints = []
    for frameA, frameB in color_framesets(...):
        try:            
            # Find corners
            cornersA, idsA, rejected = cv2.aruco.detectMarkers(frameA, charucoDict)
            cornersB, idsB, rejected = cv2.aruco.detectMarkers(frameB, charucoDict)
            if not cornersA or not cornersB: raise Exception("No markers detected")

            retA, cornersA, idsA = cv2.aruco.interpolateCornersCharuco(cornersA, idsA, frameA, charucoBoard)
            retB, cornersB, idsB = cv2.aruco.interpolateCornersCharuco(cornersB, idsB, frameB, charucoBoard)
            if not retA or not retB: raise Exception("Can't interpolate corners")


            # Find common points in both frames (is there a nicer way?)
            objPtsA, imgPtsA = cv2.aruco.getBoardObjectAndImagePoints(charucoBoard, cornersA, idsA)
            objPtsB, imgPtsB = cv2.aruco.getBoardObjectAndImagePoints(charucoBoard, cornersB, idsB)

            # Create dictionary for each frame objectPoint:imagePoint
            ptsA = {tuple(a):tuple(b) for a, b in zip(objPtsA[:,0], imgPtsA[:,0])}
            ptsB = {tuple(a):tuple(b) for a, b in zip(objPtsB[:,0], imgPtsB[:,0])}
            common = set(ptsA.keys()) & set(ptsB.keys())    # intersection between obj points
            
            for objP in common:
                objectPoints.append(np.reshape(objP, (1, 3)))
                imagePointsA.append(np.reshape(ptsA[objP], (1, 2)))
                imagePointsB.append(np.reshape(ptsB[objP], (1, 2)))

        except Exception as e:
            print(f"Skipped frame: {e}")
            continue

    result = cv2.stereoCalibrateExtended(objectPoints, imagePointsA, imagePointsB, intrA, distA, intrB, distB, (848, 480), flags=cv2.CALIB_FIX_INTRINSIC+cv2.CALIB_USE_EXTRINSIC_GUESS)

user_15585833 · Answer

I have just made something similar earlier today. I assume you solved at least part of your issue, since you closed the mentioned bug. In any case, it seems to me that the issue is that you are passing an array of points, while it should be an array of array of points (an array of point for each frame with sufficient data).

On a related note, cv2.aruco.getBoardObjectAndImagePoints is probably not what you are looking for, with cornersA and cornersB already being image points (of chessboard pattern corners), and object points (position of the chessboard pattern corners) being computable from the aruco marker ids, while getBoardObjectAndImagePoints is about the aruco marker corners as far as I can tell.

Internally, cv2.aruco.calibrateCameraCharuco simply calls cv2.calibrateCamera with the passed corners as image points, and the object points computed from the passed aruco IDs. Unfortunately, getting the object point from an aruco ID isn't exposed in the API, but it's pretty easy to compute: https://github.com/opencv/opencv_contrib/blob/master/modules/aruco/src/charuco.cpp#L157-L166

OpenCV stereoCalibration of two cameras using ChArUco

1 Answers