Automatic Topic Labeling Evaluation metric

user_11531123 0
Viewed 482

I am trying to do a topic labeling problem on a large dataset of research papers. The idea is that I can give each paper a few relevant labels.

I have 2 questions.

I know you can do topic modeling in a variety of ways like using LDA and NMF, but what can you do to later extract possible labels from those topics?

Also, assuming I have extracted a bunch of labels, how can I mathematically estimate their accuracy? Is there some kind of metric available that can determine say, the variance of the information explained by a label in a document, or something along those lines? How would I evaluate my labels without a large group of humans doing qualitative analysis?

topic-modeling nlp python
1 Answers

The most simple way is to use the top k words as the labels. More complicated methods include candidate label generation and candidate label ranking. Many related papers talking about this topic:

  1. Aletras, Nikolaos, and Mark Stevenson. "Labelling topics using unsupervised graph-based methods." ACL. 2014
  2. Bhatia, Shraey, Jey Han Lau, and Timothy Baldwin. "Automatic labelling of topics with neural embeddings." COLING (2016).
  3. Hingmire, Swapnil, et al. "Document classification by topic labeling." SIGIR. 2013

All the above papers have sections discussing how to evaluate the labels.

user_47947010
Related