So I'm trying to find similar sentences in a moderately large file with 60000 rows. Now to accomplish this, I first created sentence encodings of each row using google universal sentence encoder. Then I use this to compare cosine similarity and find similar sentences
module_url = "https://tfhub.dev/google/universal-sentence-encoder/4"
model = hub.load(module_url)
sentence_embeddings = model(sentences)
def cosine(u, v):
return numpy.dot(u, v) / (numpy.linalg.norm(u) * numpy.linalg.norm(v))
for idx,query in list(enumerate(sentences)):
for idx2,sente in enumerate(sentences):
if idx1 == idx2:
continu
sim = cosine(sentence_embeddings[idx], sentence_embeddings[idx2])
if sim >= .80:
# store in output
So with 60000^2 operations of cosine similarity, it takes days on my device to execute this code. Is there a way I can do this faster. I understand that this is probably as fast as I can get with python so if the solution involves using some other language, im open to it as well
Thanks a ton