How to list all documents/words per topic in bert topic modelling?

user_15563833 0
Viewed 167

I read the docs, but i can see the topics only show 3 or 4 documents per topic whereas the count is 2000+, is there a way i can see all the assigned documents, instead of three/four documents per topic? For example: i want to see all the 2555 documents in the below picture, and get all words under the name column, not just first 3, 4 words I tried many things, but it doesn't workenter image description here

sentence-transformers bert-language-model dataframe python
1 Answers

as I understood you want to see n_words of the model and also documents representing the specific topics. First of all, you can list all topics b following code:

import pandas as pd
with pd.option_context('display.max_rows', None,
                       'display.max_columns', None,
                       'display.precision', 3,
                       ):
    print(freq1)

with this, you will see all the topics you have on your model.

In order to get n_words from topic 3 you can run this command:

model1.get_topic(3)

and you will get

[('hood', 0.08646080854070591),
 ('fort', 0.07903592661513956),
 ('terrorist', 0.04050269806508548),
 ('muslim', 0.0404965762204116),
 ('ft', 0.04046989026050265),
 ('militari', 0.03581303765985982),
 ('armi', 0.025703775870144486),
 ('base', 0.025620172464129863),
 ('islam', 0.024491265378088094),
 ('attack', 0.02280540444895898)]

output like this, n_words with its c-TF-IDF score.

you can also get all topics with their own n_words by running:

model1.get_topics()

if you want to get documents that represents topics you can run:

model1.get_representative_docs(3)

where the output will be like:

['wouldnt fort hood islam terror attack radic jihadist',
 'wasnt fort hood consid terrorist attack',
 'kind gun use fort hood']
user_191258400
Related