How do you find a specific marker in a distorted image?

How do you find a specific point in a distorted image?

On a rectangle 3 circles are given, whose coordinates can be found with the function HoughCircles with OpenCV. This is implemented with the following code which works without problems:

import cv2 as cv
import numpy as np

fig1 = cv.imread('shapes.jpg', cv.IMREAD_COLOR)

fig = cv.cvtColor(fig1, cv.COLOR_BGR2GRAY)
fig = cv.medianBlur(fig, 5)

rows = fig.shape[0]
circles = cv.HoughCircles(fig, cv.HOUGH_GRADIENT, 1, rows / 8,
                           param1=100, param2=30,
                           minRadius=1, maxRadius=40)

circles = np.uint16(np.around(circles))

for i in circles[0,:]:
    cv.circle(fig1, (i[0], i[1]), i[2], (255, 0, 255), 3)

if circles is None:
    print('No circle found!')

cv.imshow("detected circles", fig1)
cv.waitKey(0)

# previously measured distances
ABold=5 # cm
BCold=5 # cm
e=1 # cm
f=1 # cm

x0=float(circles[0,0,0])
y0=float(circles[0,0,1])
x1=float(circles[0,1,0])
y1=float(circles[0,1,1])
x2=float(circles[0,2,0])
y2=float(circles[0,2,1])

# calculate all distances
A=np.sqrt(((x0-x1)**2)+((y0-y1)**2))
B=np.sqrt(((x1-x2)**2)+((y1-y2)**2))
C=np.sqrt(((x2-x0)**2)+((y2-y0)**2))  

# determining the hypotenuse
if A>B and A>C:
    xA=x2
    yA=y2
    xC=x1
    yC=y1
    xB=x0
    yB=y0
    AB=C
    BC=B    
elif B>A and B>C:
    xB=x0
    yB=y0
    xA=x1
    yA=y1
    xC=x2
    yC=y2
    AB=A
    BC=C
else:
    xC=x1
    yC=y1
    xB=x2
    yB=y2
    xA=x1
    yA=y1
    AB=A
    BC=B
# new position of x-symbol
xX=(xA+e*(AB/ABold))
yX=(yA+f*(BC/BCold))
    
from matplotlib import pyplot as plt

plt.plot(xA,yA, marker="o", markersize=20, markerfacecolor="green", label="A")
plt.plot(xB,yB, marker="o", markersize=20, markerfacecolor="red", label="B")
plt.plot(xC,yC, marker="o", markersize=20, markerfacecolor="blue", label="C")
plt.plot(xX,yX, marker="x", markersize=20, markerfacecolor="black")
plt.legend()
plt.xlim([0,1200])
plt.ylim([0,1200])
plt.grid()
plt.show()

Sorry that the code is not perfect. Hopefully you can see through there. I just programmed it quickly to see where the problems can occur.

In the pictures you can see the taken image and the result generated with the code:

Picture of Points on Plane to recognize

Result Picture

The distances of all points and the marker are known. However, on the taken image they are distorted due to the recording. How is it possible to determine the position of the marker on the distorted image? The problem: the circles do not differ and are detected in a random order. So it is not directly possible to determine the new coordinates simply from the changed distances, since it is not known from which point one has to take the new distance.

Any ideas?