How to get the TF-IDF scores as well for the most important words?

user_14288137 0
Viewed 149

I am working on a project with tf-idf, I have a column (df['liststring']) in my dataframe that contains the preprocessed text (without punctuation, stop words, etc.) from my various documents.

I ran the following code, and I got the top 10 words with the highest tf-idf values but I would like to see their scores as well.

    from sklearn.feature_extraction.text import TfidfVectorizer
    tfidf = TfidfVectorizer()
    X_tfidf = tfidf.fit_transform(df['liststring']).toarray()
    vocab = tfidf.vocabulary_
    reverse_vocab = {v:k for k,v in vocab.items()}
    feature_names = tfidf.get_feature_names()
    df_tfidf = pd.DataFrame(X_tfidf, columns = feature_names)
    idx = X_tfidf.argsort(axis=1)
    tfidf_max10 = idx[:,-10:]
    df_tfidf['top10'] = [[reverse_vocab.get(item) for item in row] for row in tfidf_max10 ]
    
df_tfidf['top10']

0      [kind, pose, world, preventive, sufficient, ke...
1      [mode, california, diseases, evidence, zoonoti...
2      [researcher, commentary, allegranzi, say, mora...
3      [carry, mild, man, whatever, suffering, downpl...
4      [region, service, almost, wednesday, detect, f...
                             ...                        
754    [americans, plan, year, black, online, shop, s...
755    [relate, manor, tuesday, death, portobello, ce...
756    [one, october, eight, exist, transmit, cluster...
757    [wolfe, shelter, county, resident, cupertino, ...
758    [firework, year, blasio, day, marching, reimag...

If we take the first row as an example, instead of [kind, pose, world, preventive, sufficient, ke...], I would like to get the output look like [kind:0.2, pose:0.3, world:0.4, preventive:0.5, sufficient:0.6, ke...]

scikit-learn tf-idf nlp python
1 Answers
df_tfidf['top10'] = [[(reverse_vocab.get(item), X_tfidf[i, item])  for item in row] 
                     for i, row in enumerate(tfidf_max10) ]

Test case:

df = pd.DataFrame(
    {'liststring': ['this is a cat', 'that is a dog', "a apple on the tree"]}
)
tfidf = TfidfVectorizer()
X_tfidf = tfidf.fit_transform(df['liststring']).toarray()
vocab = tfidf.vocabulary_
reverse_vocab = {v:k for k,v in vocab.items()}
feature_names = tfidf.get_feature_names()
df_tfidf = pd.DataFrame(X_tfidf, columns = feature_names)
idx = X_tfidf.argsort(axis=1)
tfidf_max2 = idx[:,-2:]
print ([[(reverse_vocab.get(item), X_tfidf[i, item])  for item in row] 
                     for i, row in enumerate(tfidf_max2) ])

Output:

[[('cat', 0.6227660078332259), ('this', 0.6227660078332259)],
 [('dog', 0.6227660078332259), ('that', 0.6227660078332259)], 
 [('the', 0.5), ('tree', 0.5)]]
user_4239260
Related