I am trying to convert a 2D pixel point into a 3D world position on a plane where Z = 0. Similar to this question on the opencv forum. Check it out for a very nice illustration of the setup.
Some explination. To convert between image coordinates and world coordinates a homography matrix is commonly used. The Homography matrix consist of an internal matrix and an external matrix.
Pix Internal External Position
[u] [ f 0 Px 0] [ R11 R12 R13 Tx] [x]
[v] = [ 0 f Py 0] [ R21 R22 R23 Ty] [y]
[1] [ 0 0 0 0] [ R31 R32 R33 Tz] [z]
[ 0 0 0 1 ] [1]
[Homograpy] = [Interal][External]
Using the homography matrix it is possible to convert from world position to camera position. Then inversing the homograhy matrix it is possible to convert pixel position to world position. As long as Z is set as depth is lost when converting to pixels, so we insert a new value.
How to combine internal and external matrices into a homography matrix and use the homography matrix can be found several places. Some C++ Github code for this can be found here.
Not the internal matrix has to be using OpenCv camera calibration which is its own topic. Now I have already done this so I will be skipping this topic.
The external matrix consists of a rotation matrix [R] and a translation vector |t|. In other OpenCv questions the rotation matrix and translation vector is calculated as follows:
solvePnP(world_points, image_points, cameraMatrix, distCoeffs, rotationVector, translationVector);
Rodrigues(rotationVector, rotationMatrix);
Now I my use case the camera position and rotation are not fixed. So it would be a lot of work to create mapped world and image positions for each pose. However, I would know the camera location and rotation in world space. So is it possible to use these to construct the extrinsic matrix?
So of the reading that I have done have hinted at that it should be possible: The formulas here, suggests that is possible to calculate extrinsic matrix as follows:
[External] = [R|t] = [R| -RC]
In this equation C is the position of the camera expressed in world coordinates. So with C known I only need R to find the translation vector.
Now this article here here inserts the [Roll, Pitch, Yaw] (θ, φ, ψ) in radians into x,y,z-axis rotation matrices to create the final combined rotation matrix.
Now I have tried tried doing this before but never figured out the order of rotation. Since that matters a lot when it comes to rotation matrices.
For example would the order be the same as above if the camera is rotated -90* around the Z-axis and then 90* around the X-axis, to get to the position as shown in the image bellow? Or does the order not matter in this case, as we are not rotating the object but describing its rotation?
One other option could be to instead create the rotation vector then pass it thought the Rodrigues to then get the rotation matrix. As the other OpenCv examples do. I would then just need to find a way to express the rotation vector.
