Huggingface Transformers FAISS index scores

user_4106227 0
Viewed 91

Huggingface transformers library has a pretty awesome feature: it can create a FAISS index on embeddings dataset which allows searching for the nearest neighbors.

train_ds['train'].add_faiss_index("embedding")
scores, sample = train_ds.get_nearest_examples("embedding", query_embedding, k=10)


I'm trying to understand the significance of the scores and the intuition behind it. For example if we were to relate cosine similarity and the faiss search score this is what we get:

from scipy.spatial.distance import cosine
print("cosine sim", "faiss score")
for i in range(10):
  distance = cosine(query_embedding, sample["embedding"][i])
  print(np.round(1-distance, 3), scores[i])


we get this:

cosine sim   faiss score
0.9983       75.67109
0.9961       149.42262
0.9969       169.43077
0.9939       243.45598
0.9914       284.8134
0.9963       309.41052
0.9934       327.74158
0.9898       330.72858
0.9897       337.12408
0.99         337.60126


As you can see the cosine similarity is pretty much uniform and is very close to 1. However, the faiss search scores vary widely. I'm trying to understand what do these numbers represent and how are they calculated. Can they be used to return search results above some treshold? Documentation doesn't cover this unfortunately

faiss cosine-similarity huggingface-transformers
1 Answers

FAISS uses binning and PQ (Product Quantization) to yield approximate answers quickly and requiring considerably less memory. So the score might bounce around because of this approximation. It's not even guaranteed to find all KNN because of the approximation (due to sampling of only some bins, I think).

So yes, you can use a cutoff if you want, realizing the clever shortcuts that FAISS is taking won't ever yield something that's equivalent to cosine similarity. But cosine similarity cannot do the tasks that FAISS can do.

user_6073050
Related