Speeding up list processing in Python

user_13622500 0
Viewed 86

I have two lists list1 and list2, each of size 5000 with every entry of the lists being a numpy.array. I want to calculate the squares of the Euclidean distances between the elements of the lists in a fast and efficient way, i.e. I need to calculate sum((list1[i]-list2[j])**2) for every combination of i and j, which are thus in total 2,500,000 combinations. I currently did so by running a double loop and writing every result into a 2d numpy.array by means of 

result[i,j] = sum((list1[i]-list2[j])**2)

but still need on my computer about 4 minutes of time. I was wondering, whether any tricks could be used to further speed the calculation up.

scipy list numpy python
1 Answers

If you insist on numpy (assuming your inner arrays are 1-D):

dist_mat = ((list1[:,None,:]-list2[:,:])**2).sum(2)

Note that according to your definition of distance in question, this is square of Euclidean distances. If you want the distance itself, simply take square root of this.
Otherwise, I would prefer @Quang's comment:

from scipy.spatial import distance_matrix
dist_mat = distance_matrix(list1, list2)
user_49759810
Related