choose the best Coherence Score for LDA model

Question

I am using python Gensim package to build LDA model (https://www.machinelearningplus.com/nlp/topic-modeling-gensim-python/#:~:text=Topic%20Modeling%20with%20Gensim%20(Python)&text=Topic%20Modeling%20is%20a%20technique,in%20the%20Python's%20Gensim%20package)

To choose the best number of Topics in the LDA, I calculated the Coherence score for (1-20)topics and then visulaize it

    def compute_coherence_values(dictionary, corpus, texts, limit, start=1, step=1):
    coherence_values = []
    model_list = []
    for num_topics in range(start, limit, step):
        lda_model_coh = gensim.models.ldamodel.LdaModel(corpus=corpus, num_topics=num_topics, id2word=id2word, per_word_topics=True,update_every=1,
                                           chunksize=100,random_state=80,
                                           passes=10,
                                           alpha='auto')
        model_list.append(lda_model_coh)
        coherence_model_lda = CoherenceModel(model=lda_model_coh, texts=data_words_nostops, dictionary=id2word, coherence='c_v')
        coherence_values.append(coherence_model_lda.get_coherence())

    return model_list, coherence_values

model_list, coherence_values = compute_coherence_values(dictionary=id2word, corpus=corpus, texts=data_words_nostops, start=1, limit=21, step=1)
print(coherence_values)

the out put is :

[0.6110807023750182, 0.623346262237542, 0.611190819343431, 0.6150879617345366, 0.6661056841233617, 0.6460622418348893, 0.6684570240561849, 0.6603704258720786, 0.6781376351229919, 0.6686810583507139, 0.6704931154541898, 0.6209832171172912, 0.6223242456220992, 0.583528787158143, 0.5672411886488239, 0.5485767400671002, 0.5603438856884889, 0.538775236148759, 0.5424604528457801, 0.536498799229393]

coherence score / topic numbers ][1]

As the chart shows, The coherence score value is highest score at the value 10. But when i visualize it using the intertopic Distance maps i found the topics are crowded and the overlaps between the topics is huge. So i visualize it using 5 and 7 topics. 2 I am not sure which number of topics is better to choose. My question is, How to choose the best coherence value. a scintfic reference recommendation will be a good help for me.
[1]: https://i.stack.imgur.com/ltqrC.png

choose the best Coherence Score for LDA model

0 Answers