Finding the characteristics of a hand written Arrows with opencv

Question

I'm trying to retrieve the orientation of a hand written arrows: after removing shadows and applying binarization and dilating the lines, here are the images:

Now I'd like to get the orientation of the arrow so I have tried using HoughLines,

lines = cv2.HoughLines(edges, rho=1, theta=np.pi / 180, threshold=20)

But is seems it generates too many lines (around 54 lines), I'd like it to generate only 3 lines so I would be able to find the intersection of those lines. I can group the lines into groups of similar angle (+/-20 degrees) and then average the angle. but I'm not sure what should be rho of an average line, can somebody please give a simple example?

Is there any other approach which may be more accurate?

I'll be glad to hear, thank you all

user_4734439 · Answer

I suggest a different approach. In summary, the approach goes as follows (made it in a hurry, might need some tuning):

Find the center of the minimum area rectangle (rotated rectangle) that encloses the whole arrow. (The circle drawn in the third image)
Find the center of gravity for all white points. It will be shifted a bit towards the actual head of the arrow. (Drawn in 4th pic as the origin of the eigenvector)
Find eigenvectors for all white points.
Find the displacement vector (the center of gravity - the center of the rotated rectangle)

Now:

Arrow angle(unoriented): is the angle of the first eigenvector
Arrow direction: is the sign of the dot product of (the first eigenvector and the centers' displacement vector)

Code:

Parts related to PCA are inspired by and mostly copied from this. I only made a minor change to the "getOrientation" method, added the following lines before it returns

angle = (angle - math.pi) * 180 / math.pi
return angle, (mean[0,0]), (mean[0,1]), p1

Code implementing the logic above:

#threshold
_, img = cv2.threshold(img, 128, 255, cv2.THRESH_OTSU)
imshow(img)

#close the image to make sure the contour is connected)
st_el = cv2.getStructuringElement(cv2.MORPH_RECT, (5, 5))
img = cv2.morphologyEx(img, cv2.MORPH_CLOSE, st_el)
imshow(img)

#get white points
pnts = cv2.findNonZero(img)

#min area rect
rect_center = cv2.minAreaRect(pnts)[0]

#draw rect center
cv2.circle(img, (int(rect_center[0]), int(rect_center[1])), 3, 128, -1)
imshow(img)

angle, pca_center, eigen_vec = getOrientation(pnts, img)

cc_vec = (rect_center[0] - pca_center[0], rect_center[1] - pca_center[1])
dot_product = cc_vec[0] * eigen_vec[0] + cc_vec[1] * eigen_vec[1]
if dot_product > 0:
    angle *= -1

print ("Angle = ", angle)
imshow(img)

Edit

I suggest a simpler method. This new method does not depend on PCA for finding the unoriented angle [0 - 180]. Instead, uses the min area rectangle angle immediately. And uses the contour momentum for finding the center of gravity.

Simpler Method Code:

#get white points
pnts = cv2.findNonZero(img)

#min area rect
rect_center, size,  angle = cv2.minAreaRect(pnts)

#simple fix for angle to make it in [0, 180]
angle = abs(angle)
if size[0] < size[1]:
    angle += 90

#find center of gravity    
M = cv2.moments(img)
gravity_center = (M["m10"] / M["m00"], M["m01"] / M["m00"])

#rot rect vec based on angle
angle_unit_vec = (math.cos(angle * 180 / math.pi), math.sin(angle * 180 / math.pi))

#cc_vec = gravity center - rect center
cc_vec = (gravity_center[0] - rect_center[0], gravity_center[1] - rect_center[1])

#if dot product is negative add 180 -> angle between [0, 360]
dot_product = cc_vec[0] * angle_unit_vec[0] + cc_vec[1] * angle_unit_vec[1]
angle += (dot_product < 0) * 180

#draw rect center
cv2.circle(img, (int(rect_center[0]), int(rect_center[1])), 3, 128, -1)
cv2.circle(img, (int(gravity_center[0]), int(gravity_center[1])), 3, 20, -1)

imshow(img)
print ("Angle = ", angle)

Edit2: This edit includes these changes:

Use cv2.fitLine() and use the fitted line angle for orientation.
Replace angle_unit_vec with a vector that has the gravity center as the origin and goes parallel to the fitted line.

Code

#get white points
pnts = cv2.findNonZero(img)
#min area rect
rect_center, size,  angle = cv2.minAreaRect(pnts)

#fit line to get angle
[vx, vy, x, y] =cv2.fitLine(pnts, cv2.DIST_L12, 0, 0.01, 0.01)
angle = (math.atan2(vy, -vx)) * 180 / math.pi

M = cv2.moments(img)
gravity_center = (M["m10"] / M["m00"], M["m01"] / M["m00"])

angle_vec = (int(gravity_center[0] + 100 * vx), int(gravity_center[1] + 100 * vy))

#cc_vec = gravity center - rect center
cc_vec = (gravity_center[0] - rect_center[0], gravity_center[1] - rect_center[1])

#if dot product is positive add 180 -> angle between [0, 360]
dot_product = cc_vec[0] * angle_vec[0] + cc_vec[1] * angle_vec[1]
angle += (dot_product > 0) * 180


angle += (angle < 0) * 360


#draw rect center
cv2.circle(img, (int(rect_center[0]), int(rect_center[1])), 3, 128, -1)
cv2.circle(img, (int(gravity_center[0]), int(gravity_center[1])), 3, 20, -1)

imshow(img)
print ("Angle = ", angle)

Output: Using code from edit2:

First image:

Second image:

Third image:

Finding the characteristics of a hand written Arrows with opencv

1 Answers