Computing Skip-gram Frequency with countVectorizer

user_10168484 0
Viewed 562

I'm trying to compute the most frequent skip-grams in a text file. I'm using nltk's skipgram and scikit-learn's countVectorizer, but it gives me a list of distinct skip-grams. Hence, when I put them in a dictionary to count them, I get frequency = 1 for every skip-gram.

I believe this is because I'm using the vectorizer.vocabulary_ method, which skips the repeating skip-grams.

I'm using this code https://github.com/nltk/nltk/issues/1428#issuecomment-231647710

In this original code they weren't trying to compute the frequency, so distinct skip-grams (vocabulary) was fine. In my case, how can I change the code so that I get the comprehensive list of all the skip-grams generated by countVectorizer?

import functools
from nltk.util import skipgrams
from nltk import word_tokenize
from sklearn.feature_extraction.text import CountVectorizer

text = [word_tokenize(line.strip()) for line in open('test.txt', 'r')]
skipper = functools.partial(skipgrams, n=2, k=2)
vectorizer = CountVectorizer(analyzer=skipper)
vectorizer.fit(text)
vectorizer.vocabulary_

dict = {}
dict = vectorizer.vocabulary_

def getList(dict): 
    return dict.keys() #get all the skip-grams

#store all skip-grams in a list to count their frequencies
newlist = []
for key in getList(dict):
  newlist.append(key) 

#count frequency of items in list
def count(listOfTuple):       
    count_map = {} 
    for i in listOfTuple: 
        count_map[i] = count_map.get(i, 0) +1
    return count_map 

d = count(newlist)
print(d)

For example, if I have a text consisting of two strings "i love apple" and "i love watermelon" print(d) should give:

('i', 'love'):2
('i', 'apple'):1
('i', 'watermelon'):1

However right now I am getting 1 everywhere.

Any help would be greatly appreciated!!

scikit-learn n-gram nltk nlp python
1 Answers

You identified the problem well, you should not be using vectorizer.vocabulary_. So you can keep this:

import functools
from nltk.util import skipgrams
from nltk import word_tokenize
from sklearn.feature_extraction.text import CountVectorizer

text = [word_tokenize(line.strip()) for line in ["I love apple","I love 
pineapple"]]
skipper = functools.partial(skipgrams, n=2, k=2)
vectorizer = CountVectorizer(analyzer=skipper)
vectorizer.fit(text)

But here, use your vectorizer object to actually transform your text into its vectorized version:

vectorized_text = vectorizer.transform(text)
print(dict(zip(vectorizer.get_feature_names(),vectorized_text.toarray().sum(axis = 0))))

Then you will get, as expected:

>>> {('I', 'apple'): 1,
     ('I', 'love'): 2,
     ('I', 'pineapple'): 1,
     ('love', 'apple'): 1,
     ('love', 'pineapple'): 1}
user_109566060
Related