I am working on clustering a data set by applying k means. My ultimate goal is to use top terms of cluster centroid to label my data. As for the data I use scraped TripAdvisor user reviews. So that is the background. My issue when I print the top 100 terms of cluster centroids, it shows the same word more than once.
Here is my code
all_txt_files =[]
for file in Path("txt").rglob("*.txt"):
all_txt_files.append(file.parent / file.name)
# counts the length of the list
n_files = len(all_txt_files)
print(n_files)
reviews = []
titles = []
for txt_file in all_txt_files:
with open(txt_file ,encoding="utf8") as f:
txt_file_as_string = f.read()
file_name = f.name
reviews.append(txt_file_as_string)
titles.append(file_name)
clean_reviews = []
Nouns = []
ranks = []
temp=[]
totalvocab_stemmed = []
totalvocab_tokenized = []
Resting_Places =['motel','dome','restroom','hotel','villa','restaurant','lodge']
Label_Resting_Places=[]
count=0
for i in reviews:
allwords_stemmed = tokenize_and_stem(i) #for each item in 'synopses', tokenize/stem
totalvocab_stemmed.extend(allwords_stemmed) #extend the 'totalvocab_stemmed' list
allwords_tokenized = tokenize_only(i)
totalvocab_tokenized.extend(allwords_tokenized)
vocab_frame = pd.DataFrame({'words': totalvocab_tokenized}, index = totalvocab_stemmed)
tfidf_vectorizer = TfidfVectorizer(max_df=0.65, max_features=200000,
min_df=0.04, stop_words='english',
use_idf=True, tokenizer=tokenize_and_stem, ngram_range=(1,3))
tfidf_matrix = tfidf_vectorizer.fit_transform(reviews)
terms = tfidf_vectorizer.get_feature_names()
vocab = tfidf_vectorizer.vocabulary_
dist = 1 - cosine_similarity(tfidf_matrix)
num_clusters = 4
km = KMeans(n_clusters=num_clusters)
km.fit(tfidf_matrix)
clusters = km.labels_.tolist()
joblib.dump(km, 'doc_cluster.pkl')
km = joblib.load('doc_cluster.pkl')
clusters = km.labels_.tolist()
user_review = { 'title': titles, 'rank': ranks, 'reviews': reviews, 'cluster': clusters }
frame = pd.DataFrame(user_review, index = [clusters] , columns = ['rank', 'title', 'cluster'])
grouped = frame['rank'].groupby(frame['cluster'])
print("Top terms per cluster:")
print()
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
for i in range(num_clusters):
print("Cluster %d words:" % i, end='')
for ind in order_centroids[i, :100]:
feature=vocab_frame.loc[terms[ind].split(' ')].values.tolist()[0][0]
print(' %s' % vocab_frame.loc[terms[ind].split(' ')].values.tolist()[0][0].encode('utf-8', 'ignore'), end=',')
if( feature in Resting_Places):
temp.append(feature)
print()
Label_Resting_Places.append(temp)
temp=[]
print()
print(Label_Resting_Places)
I exclude some text preprocessing parts from the code otherwise it will be a lengthy code. So here are the top terms of first cluster centroid.
Cluster 0 words: b'gym', b'ambiance', b'paris', b'terrace', b'decor', b'hotel', b'ocean', b'leeu', b'just', b'patio', b'race', b'husband', b'tennis', b'center', b'island', b'mountain', b'hiking', b'ottawa', b'westin', b'dinner', b'tennis', b'trails', b'hotel', b'wine', b'room', b'pagoda', b'surfers', b'wave', b'spa', b'service', b'pools', b'hotel', b'cliffs', b'hotel', b'chaise', b'fairmont', b'kl', b'hotel', b'kangaroos', b'infinity', b'infinity', b'leeu', b'standard', b'fact', b'hotel', b'rer', b'paris', b'hotel', b'sand', b'waterfall', b'lamb', b'appetizer', b'sydney', b'cape', b'tuna', b'price', b'restaurant', b'harbour', b'beet', b'sail', b'glacier', b'balloon', b'sauna', b'hotel', b'gym', b'service', b'size', b'expectations', b'bistro', b'foods', b'location', b'place', b'fremantle', b'club', b'med', b'ficks', b'adelaide', b'cove', b'whistler', b'ambiance', b'montreal', b'paris', b'foods', b'massy', b'restaurant', b'sugar', b'loudness', b'aloft', b'bit', b'california', b'edge', b'infinity', b'hotel', b'kong', b'things', b'filet', b'hong', b'neighborhood', b'hong', b'foods',
As you can see it shows the word 'hotel' more than once. Can someone please help me solve this issue? It will be really helpful.