Why it is so slow when calculating the coherence score for LDA using gensim

user_11683888 0
Viewed 1681

I am new to LDA and when I am calculating the coherence score for my LDA model using gensim CoherenceModel, it takes extremely long time to run. However, the training part is relatively fast, and in a reasonable time. I wonder if this is because of my data size(about 250000 long text) and are there any ways to speed up this process? Thanks

This is my code, which is exactly the same as the tutorial

from gensim.models import CoherenceModel
coherence_model_lda = CoherenceModel(model=lda_model_tfidf, texts=LDA_, dictionary=dictionary, coherence='c_v')
coherence_lda = coherence_model_lda.get_coherence()
print('\nCoherence Score: ', coherence_lda)
lda gensim nlp
1 Answers

The 'c_v' coherence measure is known to be the slowest method but gets the best results. You can try 'u_mass' for the fastest performance.

Note that you'll only need the model, corpus, and coherence parameters for 'u_mass'.

You can read more on the difference between the two methods here: http://palmetto.aksw.org/palmetto-webapp/

Admin1
Related