How to sort YOLOv4 bounding box?

I have trained the yolov4 from alexeyab darknet repo to detect characters in a number plate. It segments the character correctly but the bounding boxes are in random order. How can I sort the bounding box from top left to bottom right for image like this: (This is not actual image used but this is a photoshoped image for sample Nepali License Number Plate because of confidential data)

I've tried: (from pyimagesearch)

def sort_bbox(bbox, method="left-to-right"):
    # initialize the reverse flag and sort index
    reverse = False
    i = 0
    # handle if we need to sort in reverse
    if method == "right-to-left" or method == "bottom-to-top":
        reverse = True
    # handle if we are sorting against the y-coordinate rather than
    # the x-coordinate of the bounding box
    if method == "top-to-bottom" or method == "bottom-to-top":
        i = 1
    # construct the list of bounding boxes and sort them from top to
    # bottom
    boundingBoxes = sorted(bbox, key=lambda b: b[1], reverse=reverse)
    # return the list of sorted contours and bounding boxes
    return boundingBoxes

but didn't sort the bounding boxes. It's still in random order.

I have bounding box from yolov4 detection like this: unsorted bounding boxes in xywh: [[50, 12, 15, 18], [66, 10, 15, 19], [87, 10, 19, 20], [21, 12, 24, 19], [51, 12, 15, 17], [51, 12, 15, 18], [66, 12, 15, 18], [86, 11, 19, 19], [39, 32, 27, 29], [68, 33, 28, 27], [97, 31, 28, 30], [12, 37, 24, 25], [11, 35, 25, 27], [40, 34, 27, 28], [68, 33, 27, 27], [97, 33, 28, 28]]

and from above sorting code: [[66, 10, 15, 19], [87, 10, 19, 20], [86, 11, 19, 19], [50, 12, 15, 18], [21, 12, 24, 19], [51, 12, 15, 17], [51, 12, 15, 18], [66, 12, 15, 18], [97, 31, 28, 30], [39, 32, 27, 29], [68, 33, 28, 27], [68, 33, 27, 27], [97, 33, 28, 28], [40, 34, 27, 28], [11, 35, 25, 27], [12, 37, 24, 25]]

What I want is the bounding box of: बा २ प ८ ८ ८ ८

Any Help will be very much appreciated.