Convert Topic-name and Description in pandas into Topic-Name, Words and Frequency

user_10795530 0
Viewed 82

I have a pandas data-frame consist of Topic-Name,Description

Original Data

I am trying to convert it into Topic-Name, Word (appeared in Description Column) and its Frequency

Expected Result

I have tried this but it only gave me word and frequency for Descriptions, not on Topic-Name Level.

import nltk
top_N = 4
a = df['ParsedText'].str.cat(sep=' ')
nltk.download('punkt')
words = nltk.tokenize.word_tokenize(a)
word_dist = nltk.FreqDist(words)
rslt = pd.DataFrame(word_dist.most_common(top_N),
                columns=['Word', 'Frequency'])
print(rslt)
  • Parsed Text is a Description Column
pandas nltk python
1 Answers

You can try:

(df.assign(Description=df.Description.str.split()).explode()
   .groupby('TopicName')['Description'].value_counts()
   .reset_index(name='Frequency')
)
user_42384080
Related