Get clusters of words using Kmeans and TF-IDF

user_11170350 0
Viewed 126

I am trying to clusters text words. Let suppose I have a list of text

text=["WhatsApp extends 'confusing' update deadline",
"India begins world's biggest Covid vaccine drive",
"Nepali climbers make history with K2 winter summit"]

I implemented TF-IDF on this data

vec = TfidfVectorizer()
feat = vec .fit_transform(text)

After that, I applied Kmeans

kmeans = KMeans(n_clusters=num).fit(feat)

The thing I am confused about is how I get clusters of words such as

cluster 0  
WhatsApp, update,biggest   
cluster 1  
history,biggest ,world's   
etc.
scikit-learn tf-idf k-means python
1 Answers

You can use the get_feature_names() method from the TfidfVectorizer class with the predictions from KMeans to inspect the words in each cluster.

Here's a minimal example with two clusters and the three sentence provided by you:

import numpy as np
from sklearn.cluster import KMeans
from sklearn.feature_extraction.text import TfidfVectorizer

text = ["WhatsApp extends 'confusing' update deadline",
        "India begins world's biggest Covid vaccine drive",
        "Nepali climbers make history with K2 winter summit"]

vec = TfidfVectorizer()
feat = vec.fit_transform(text)
kmeans = KMeans(2).fit(feat)
pred = kmeans.predict(feat)

for i in range(2):
    print(f"Cluster #{i}:")
    words = []
    for sentence in np.array(text)[pred==i]:
        words += [fn for fn in vec.get_feature_names() if fn in sentence]
    print(words)

Result:

Cluster #0:
['confusing', 'deadline', 'extends', 'update', 'begins', 'biggest', 'drive', 'vaccine', 'world']
Cluster #1:
['climbers', 'history', 'make', 'summit', 'winter', 'with']
user_112208840
Related