How can I merge cv2 rectangle bounding boxes into polygons? (Not by overlap/threshold)

user_15405910 0
Viewed 534

I have multiple rectangle bounding boxes that I know are part of the same object (parts of a newspaper article), as in the first image. I'm trying to work out a way to merge these into one polygon bounding box, for the whole article, as in the second image.

I have seen lots of solutions based on merging overlapping bounding boxes, but I don't care if they're overlapping or not - I already know they're part of the same article. In some cases the headline is quite far away (for example above a picture), so solutions based on padding don't work either.

I feel like there should be a cv2 function that does this, but if there is, I am missing it. Any suggestions would be super helpful.

Code to create these two images:

# Individual bounding boxes

image_0 = cv2.imread('63976500-anderson-herald-bulletin-Jun-18-1968-p-64.jpg')
# Black box, to reproduce: image_0 = np.zeros((5000, 6000, 3), dtype = "uint8")

bbox_list = [[195, 3455, 633, 4213], [658, 3427, 1094, 4222], [1120, 3435, 1553, 4473], [295, 3421, 531, 3451], [201, 3313, 1548, 3409]]

for bbox in bbox_list:
    image_0 = cv2.rectangle(image_0, (bbox[0], bbox[1]), (bbox[2], bbox[3]), (0,255,0), 10)

cv2.imwrite("original_bboxes.jpg", image_0)


# Grouped bounding box

image_1 = cv2.imread('63976500-anderson-herald-bulletin-Jun-18-1968-p-64.jpg')
# Black box, to reproduce: image_1 = np.zeros((5000, 6000, 3), dtype = "uint8")

coordinates = np.array([[195,3313],[195,4222],[1120,4222],[1120,4473],[1553,4473],[1553,3313]], np.int32)

image_1 = cv2.polylines(image_1, [coordinates], True, (0,255,0), 10)

cv2.imwrite("grouped_bboxes.jpg", image_1)

Separate bounding boxes

Merged bounding boxes

opencv python
1 Answers

You could draw the convex hull of the contour points (this was drawn by hand):
(wrong convex hull image)
Then keep only the outer contour and try the polygonal approximation. I must admit that I cannot think of a smarter way of getting only vertical and horizontal lines.

As Christoph Rackwitz observed, I was wrong. Convex hull, doesn't work. Maybe an α-shape could solve the problem, but I'm not sure. An alternative approach could be to extract all the equations of the lines which define the bounding boxes, then for each point, compute the segment that connects it to the nearest line. If the line is part of the same bounding box or if the point falls outside of the bounding box, remove the segment.

It was harder than I expected, because my Python OpenCV proficiency is not enough. Nevertheless you can get nearly what you wanted in the question:

from cv2 import cv2
import numpy as np

image_0 = cv2.imread('63976500-anderson-herald-bulletin-Jun-18-1968-p-64.jpg')
bwimage = np.zeros((image_0.shape[0], image_0.shape[1]), dtype=np.uint8)

bbox_list = [[195, 3455, 633, 4213], [658, 3427, 1094, 4222], [1120, 3435, 1553, 4473], [295, 3421, 531, 3451],
             [201, 3313, 1548, 3409]]

for bbox in bbox_list:
    bwimage = cv2.rectangle(bwimage, (bbox[0], bbox[1]), (bbox[2], bbox[3]), 255, 1)

#cv2.imwrite("original_bboxes.png", image_0)

# create list of corners with bbox index
corners = []
for i, bbox in enumerate(bbox_list):
    corners.append((bbox[0], bbox[1], i))
    corners.append((bbox[0], bbox[3], i))
    corners.append((bbox[2], bbox[1], i))
    corners.append((bbox[2], bbox[3], i))

# for each corner find nearest border
for c in corners:
    min_dist = float('inf')
    min_dist_i = None
    min_dist_type = None
    for i, bb in enumerate(bbox_list):
        for side in range(4):
            thisdim = side % 2
            otherdim = 1 - thisdim
            dist = abs(c[thisdim] - bb[side])
            if dist == 0 and c[2] == i:
                pass
            elif min_dist > dist and bb[otherdim] < c[otherdim] < bb[otherdim + 2]:
                min_dist = dist
                min_dist_i = i
                min_dist_type = side

    if min_dist_i is not None:
        bb = bbox_list[min_dist_i]
        print(f"Corner ({c[0]}, {c[1]}) nearest BB: {min_dist_i} [({bb[0]}, {bb[1]})->({bb[2]}, {bb[3]})]")
        if min_dist_type % 2 == 0:
            dest = (bb[min_dist_type], c[1])
        else:
            dest = (c[0], bb[min_dist_type])
        bwimage = cv2.line(bwimage, (c[0], c[1]), dest, 255, 1)

contours, _ = cv2.findContours(image=bwimage, mode=cv2.RETR_EXTERNAL, method=cv2.CHAIN_APPROX_NONE)
image_0 = cv2.drawContours(image_0, contours, -1, (0, 255, 0), 1)

cv2.imwrite("result.png", image_0)

Here is the result: result.png

user_60703410
Related