I have mounted two cameras on different shelves of a fridge (at the bottom and top), facing each other. An object detector is fed with two images from these two streams and returns bounding boxes from them, independently.
The problem: Given bounding boxes from different viewing angles, determine their correspondence.
I know that since depth is unknown, the x, y coordinates in one camera may correspond to multiple positions in the other. This is why our solutions hitherto have been approximations that work with varying success. Naive solution has been to ignore any difference between the camera coordinates and use the Euclidean distance to get correspondence.
Another solution is to use the Fundamental Matrix, which gives a way to calculate correspondence based on epipolar geometry. This solution may be quite tedious because it requires a form of calibration and the results have not been great. It may be due to the fact that my calibration was sloppy.
The last solution, which works poorly, is to use edge och keypoint detection and match them. Since the edges, shape etc. differs quite radically, we understand why this is so.
Ultimately, I was wondering how you would have tackled the problem or if there are anything you can point me towards to get a more robust solutions.
In the left image we have the bottom view. This image has been mirrored about y-axis. The right image is from the top shelf. The object detector has detected two bounding boxes for each viewing angle, a total of four bounding boxes. These are drawn on both cameras. For example, bounding box with id 16 is detected on the top-positioned camera and is also drawn on the bottom camera to indicate its displacement. How would you go about in matching the bounding boxes which belong to the same object viewed from another viewing angle?