Unity Hololens 2: Problem with converting screen points to world points for object detection

I'm trying to implement an object detection program for the hololens 2, but I'm having a hard time converting the output 2D screen points to 3D world coordinates so that I can draw the bounding boxes.

Here are the steps for the program:

1. Take a picture
2. Send to a server for analysis
3. Receive the 4 bounding box coordinates
4. Scale the screen point coordinates to match the unity camera pixelheight and pixelwidth dimensions
5. Raycast to the centers of the boxes using the Camera.ScreenPointToRay function

When the object is at the center of the image, it works fine. So the ray is casted correctly to the center of the bounding box, but for objects that are on the right or on the left, the raycasting is off. In particular the rays are casted slightly towards the center.

• if an object is on the left, the ray is casted slightly further to the right
• if an object is on the right, the ray is casted slightly further to the left

The same for top and bottom.

Can someone help me to explain this behavior?

Thanks