Consider the following Python code:
from scipy.spatial.distance import pdist, squareform
from fuzzywuzzy import fuzz
import pandas as pd
words = pd.DataFrame({'Words': ['horse', 'dog', 'food', 'hhorse', 'doggy']})
distance_matr = pdist(words, fuzz.ratio)
The resulting distance_matr contains valuable information which strings are the most similar with the current word. However, I am struggling to do this efficiently in terms of memory and speed.
For example: I know I could just loop through every row, pick the three smallest items, like so:
min_amount = 3
similar_words = {}
for k in range(len(words)):
current_row = squareform(distance_matr)[k]
min_indices = np.argpartition(current_row,min_amount)[:min_amount]
sim_words = words.Words[min_indices]
sim_values = current_row[min_indices]
result = sorted(zip(sim_words, sim_values), key = lambda x: x[1])
similar_words[words.Words[k]] = list(result)
print(similar_words)
However, this is extremely inefficient, as it traverses the whole row, instead of only a portion. I am wondering whether there is a method that makes use of the symmetry and vectorisation to speed this up?
Goal is actually to return the three most similar words, sorted ascending. Notice, that ideally I would not like to include the diagonal, but in this implementation I think I need to increase the min_amount and then delete the first element. But this is extremely inefficient. I was wondering whether a better method exists.