Analysis of most common n-grams

user_17468902 0
Viewed 40

Good day,

I have been studying NLP and came across this code for top n-gram extraction:

def get_top_tweet_bigrams(corpus, n=None):
    vec = CountVectorizer(ngram_range=(2, 2)).fit(corpus)
    bag_of_words = vec.transform(corpus)
    sum_words = bag_of_words.sum(axis=0) 
    words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]
    words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)
    return words_freq[:n]

I have gone over this function line by line and the part i cannot figure out is this one:

[(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]

I do understand what it achieves, but what I don't understand is how. Why simply extracting idx from vec.vocabulary_.items() gives us incorrect count? And what does the matrix sum_words hold? What are those values? Thank you.

nlp python
1 Answers
  • bag_of_words is the usual 2-dimension document-ngram frequency matrix, i.e. it contains the frequency of any ngram for any document (corpus might contain any number of documents).
  • sum_words obtains the sum of the frequency across documents for every ngram. It's a single dimension array. Length is the vocabulary size, and the indexes are in the same order as in bag_of_words. It doesn't contain the ngrams themselves, of course.

Since the goal is to obtain the top frequent ngrams, we need to match every ngram to its frequency in sum_words. This is why the vocabulary (which contains the ngrams) is iterated with the ngram and the index: if only the index idx was obtained, there would be no way to know which actual ngram it represents. And of course the index is used to obtain the total frequency in sum_words. Thus words_freq is an array containing pairs (ngram, frequency) for every ngram.

The last 2 lines sort this array by decreasing frequency and extract the top n elements.

user_8919190
Related