Consider two consecutive words as one in Word Frequency

Admin 1
Viewed 167

I have this sentence:

Sentence
    Who the president of Kuala Lumpur is?

I am trying to extract all the words (tokenization)

low_case = df['Sentence'].str.lower().str.cat(sep=' ')
words = nltk.tokenize.word_tokenize(low_case)
word_dist = nltk.FreqDist(words)

example = pd.DataFrame(word_dist.most_common(1000),
                    columns=['Word', 'Freq'])

However, I would like to extract Kuala Lumpur as bi-gram, so I am considering a filter which says "if there are two consecutive words having capital letters, extract them as a unique word. So if I have this list:

    Who the president of Kuala Lumpur is?

I would have (using the code above):

Word            Freq
who               1
is                1
president         1
of                1
Kuala             1
Lumpur            1 
is                1

but I would like to have this:

Word            Freq
who               1
is                1
president         1
of                1
Kuala Lumpur      1
is                1

I think to find two consecutive capital letters I should apply the following pattern:

pattern = r"[A-Z]{2}-\d{3}-[A-Z]{2}"

o anche:

re.findall('([A-Z][\w-]*(?:\s+[A-Z][\w-]*)+)', df.Sentence.tolist())

But I do not know how to include this information in my code above.

pandas nltk python
1 Answers

You could do some pre-processing and separate the bi-grams from the rest of the sentence using re Match Objects. For example:

import re

# initialize sentence text
sentence_without_bigrams = 'Who the president of Kuala Lumpur or Other Place is?'
bigrams = []

# loop until there are no remaining bi-grams
while True:
    # find bi-grams
    match = re.search('([A-Z][\w-]*(?:\s+[A-Z][\w-]*)+)', sentence_without_bigrams)
    if match == None:
        break
    else:
        # add bi-gram to list of bi-grams
        bigrams.append(sentence_without_bigrams[match.start():match.end()])
        # remove bigram from sentence
        sentence_without_bigrams = (sentence_without_bigrams[:match.start()-1] + sentence_without_bigrams[match.end():])


print(bigrams)
>> ['Kuala Lumpur', 'Other Place']

print(sentence_without_bigrams)
>> Who the president of or is?

However, this solution falls short of your ultimate goal, since a sentence like 'Hello, Mr President Obama' would not be captured correctly (as noted here).

user_144153330
Related