Get the most distant images in a set of images with L1 norm

I have a short video sequence containing ~100 RGB images. I stored them in a numpy array, and now I would like to get the 2 most distant images according to the L1 norm.

My first idea was to browse the set, and compare every image to the others, and store every distance in a matrix, then found the max. However, it recquires 2 for loops, and I feel that there may be a more optimal way to do this.

import numpy as np
import cv2

# Store the sequence in an array
seq = []
cap =  cv2.VideoCapture('my_video.avi')
while(cap.isOpened()) :
    ret, frame = cap.read()
    if (ret == False) :
        break
    seq.append(frame)
# Possibly convert list to np.array here

# Finding the most distant images
distance_matrix = np.zeros( (len(seq), len(seq)) )
for i in range(len(seq)) :
    for j in range(i+1,len(seq)) :
        distance_matrix[i][j] = cv2.norm(seq[i], seq[j], cv2.NORM_L1)

# Get max index (method given on scipy doc)
im1, im2 = np.unravel_index(np.argmax(distance_matrix), distance_matrix.shape)
print("The most distant frames are n°", im1, "and n°", im2)

Output would be something like >>> The most distant frames are n° 0 and n° 17

It seem that we are always forced to generate every pairs at some point, but it's an heavy method. Any idea of something more efficient ?