Group similar words/synonyms in a text and find word frequency

Viewed 32

I am trying to find the count of each word in a text. But, there are words which are synonyms in it. I want to group them together and count them as one. I am doing this in python. I am currently using wordnet to find the synonyms but not sure how to go about comparing and grouping synonyms and counting them as one.

Below is the text as a string datatype. Text = "This is a big one. Quite large. It is a bad one. But it is also large. The second one is a also evil."

I have cleaned and split the words and counted them as separate as below using python:

for char in '-?.,\n':
    Text=Text.replace(char,' ')
Text = Text.lower()
# split returns a list of words delimited by sequences of whitespace (including tabs, newlines, etc, like re's \s) 
word_list = Text.split()

from collections import Counter
Counter(word_list).most_common()

df = pd.DataFrame (Counter(word_list).most_common(), columns = ['Word','count'])

The above gives me a count of each word.

But, My desired output in the form of data frame is:

    Word  Synonym      Count
0   Big   {big,large}   3
1   Bad   {bad,evil}    2
2   Quite               1
3   one                 1
.......................................and so on

Help appreciated in getting the above output.Thanks.

Here is the way I am finding synonyms:

import nltk
#nltk.download()
from nltk.corpus import wordnet
#syns = wordnet.synsets("dog")
#print(syns)


synonyms = []

for syn in wordnet.synsets("big"):
    for l in syn.lemmas():
        synonyms.append(l.name())

print(set(synonyms))
0 Answers
Related