Coherence score (u_mass) -18 is good or bad?

Viewed 8170

I read this question (Coherence score 0.4 is good or bad?) and found that the coherence score (u_mass) is from -14 to 14. But when I did my experiments, I got a score of -18 for u_mass and 0.67 for c_v. I wonder how is my u_mass score out of range (-14, 14)?

Update: I used gensim library and scanned the numbers of topics from 2 to 50. For u_mass, it starts from 0 to the lowest negative point and turn back a bit, like an upsidedown version of c_v.

3 Answers

Following what is stated here (pg 13-14), which is the same document mentioned by @Dammio in his answer the interpretation is the opposite. In the text, it says: "According to the UMASS coherence measurements, the coherence of the topics globally decreases when K increases." K is the number of topics. They continue saying: "For the analysis, we compare models with K = 6 for 40 iterations which is a local minimum, and for 10 iterations which performed better." In the figure, it can be clearly seen that it compares the local minimum which is worse with the local maximum which is more coherent. This means exactly the opposite of what is stated in the accepted answer. Besides, I found in a Github post saying exactly the same: higher values are better:Link to Github answer

Figure 4 in pdf document

According to the mathematical formula for the u_mass coherence score provided in the original paper.

If u_mass closer to value 0 means perfect coherence and it fluctuates either side of value 0 depends upon the number of topics chosen and kind of data used to perform topic clustering. The best way to judge u_mass is to plot curve between u_mass and different values of K (number of topics). Choose K with the value of u_mass close to 0.

You can refer to this link which provides python code snippet to plot curve between different values of K and c_v. Here you can replace c_v with u_mass coherence metric.

I hope this explanation helps.

Related