I am trying to implement a document similarity API using the LDA Model of Gensim. To experiment with the performance, I tried implementing it by training the LDA Model with TfIdf vectors instead of the normal BoW corpus as described in the documentation. The problem which I am facing is that while using the Similarity API of Gensim for creating the index and finding out the similarity score, what I encountered is that if I try to match the same document with itself, sometimes, the Similarity values are not ~1. The values which I get are as low as ~0.06. This does not occur ALL the time, but for some documents only. I tested this again with 229 documents matching each document with itself, and I found that 45 of the documents give results less than 0.98, sometimes giving values like 0.65, 0.41 and similar. I would like some help on this, whether I am doing something wrong or is it any other problem.
Minimal Code used for testing:
docs = [ 'Document 1 as a string', 'Document 2 as a string', 'Document 3 as a string', 'and so on.....' ]
cleaned_docs = list(map(clean_function, docs)) # Here, clean_function return tokens for each string. So, cleaned_docs is essentially a list of list of strings List[List[str]]
bow_corpus = [dictionary.doc2bow(i) for i in cleaned_docs]
tfidf_corpus = tfidf_model[bow_corpus]
lda_corpus = lda_model[tfidf_corpus]
index = Similarity(lda_corpus)
sims = index[lda_corpus] # Getting similarity for all combinations. Got a (229, 229) array for my case
final_sims = np.diag(sims) # Getting similarity with itself
print(final_sims) # Getting very low score with some docs
Output Vectors of LDAModel for 3 documents out of the 45 I got low score for reference:
[[(0, 0.17789464), (2, 0.03806097), (12, 0.2273234), (14, 0.08613937), (21, 0.13261063), (22, 0.17807047), (36, 0.058883864)],
[(1, 0.43381935), (2, 0.14317065), (3, 0.07986226), (36, 0.062136874)],
[(0, 0.32848448), (2, 0.16667062), (14, 0.0485237), (15, 0.11480027), (18, 0.086506054), (35, 0.059970867)]]