Is there any way to join multiple vector(in jpg format) images without any overlapping?

I have a puzzle in which the image is divided into 32 tiles. These tiles are shuffled. Now I want to solve this puzzle using python and OpenCV. The problem is there is no overlapping in the images.

Example

The benefit is the image has uniform pixels since it is converted from a vector image.

What I tried is to find each image on each side for a given image based on a score for each side of the given image.

What is the score? So I take one image and for one side let's say right find the difference between last column(right side) of the image and the first column(left side) of all the other images and sum that difference and call that score.

#rightScore[i] will store the index of image on right side of the image at index 'i'.
#right[i] will store the pixels of right side of image at index 'i'.
#similarly for other sides.


    rightScore = np.zeros((36,))
    topScore = np.zeros((36,))
    bottomScore = np.zeros((36,))
    leftScore = np.zeros((36,))

    for i in range(36):
        score = np.inf
        for j in range(36): 
            if i==j: 
                continue

            temp = np.sum(np.abs(np.ravel(right[i] - left[j])))
            if score > temp: 
                rightScore[i] = j
                score = temp

Now for a given image I generate score with all other images and then find the minimum score. The corresponding image with minimum score is the image that will be on the left of the given image. I do this for all side.

This method works for some images but not for all. Can anyone help?

Also, I know the final image will have 12 rows with 3 tiles each(12 * 3 ).