I am trying to find the count of each word in a text. But, there are words which are synonyms in it. I want to group them together and count them as one. I am doing this in python. I am currently using wordnet to find the synonyms but not sure how to go about comparing and grouping synonyms and counting them as one.
Below is the text as a string datatype. Text = "This is a big one. Quite large. It is a bad one. But it is also large. The second one is a also evil."
I have cleaned and split the words and counted them as separate as below using python:
for char in '-?.,\n':
Text=Text.replace(char,' ')
Text = Text.lower()
# split returns a list of words delimited by sequences of whitespace (including tabs, newlines, etc, like re's \s)
word_list = Text.split()
from collections import Counter
Counter(word_list).most_common()
df = pd.DataFrame (Counter(word_list).most_common(), columns = ['Word','count'])
The above gives me a count of each word.
But, My desired output in the form of data frame is:
Word Synonym Count
0 Big {big,large} 3
1 Bad {bad,evil} 2
2 Quite 1
3 one 1
.......................................and so on
Help appreciated in getting the above output.Thanks.
Here is the way I am finding synonyms:
import nltk
#nltk.download()
from nltk.corpus import wordnet
#syns = wordnet.synsets("dog")
#print(syns)
synonyms = []
for syn in wordnet.synsets("big"):
for l in syn.lemmas():
synonyms.append(l.name())
print(set(synonyms))