Filtering rows within a dataframe based on n-grams

Admin 1
Viewed 200

I would need to filter rows based on the most frequent n-gram, i.e. terms that are included in the n-grams list. To calculate n-grams I used the following:

bigrams = [x for l in x_train['texts'] for x in zip(l.split(" ")[:-1], l.split(" ")[1:])]

Sample of output got from the line of code above: 

[('Donald', 'Trump,'), ('president,', 'Trump'), ('US', 'election'), ...]

Then I used sklearn to find the frequency of n-grams:

from sklearn.feature_extraction.text import CountVectorizer

word_vectorizer = CountVectorizer(ngram_range=(3,3), analyzer='word')
sparse_matrix = word_vectorizer.fit_transform(x_train['texts'])
freq = sum(sparse_matrix).toarray()[0]
df=pd.DataFrame(freq, index=word_vectorizer.get_feature_names(), columns=['Frequency'])
df.sort_values(by='Frequency', ascending=False)

An example of output is: 

    Frequency
Trump won election 46
US working vaccine  45
seqirus coronavirus vitamine    45
... ...
Apple closed shops  1
still water helps   1

My original dataset (old_df) has the following columns: 

Date               Texts                                     Messages                    Other columns 
05/24/2020     The US president Donald Trump said...     Donald Trump won election ...

... 05/01/2020 Countries are looking for a vaccine... US are working on vaccine...

I would like to filter texts and messages based on the results of the most frequent n-grams above. So, for example, since the most frequent ngrams are 

Trump won election 46
US working vaccine  45

I would like to filter my dataset, focusing on texts and messages, if texts or messages columns contains the exact words/sentence in the n-grams. Do you know if it is possible and how I could do that?

scikit-learn pandas python
1 Answers

You can use the pandas.Series.str.contains function.

An example of how it will work on the Texts column:

nth_most_frequent = 20
old_df['Text_contains_ngrams'] = 0
Texts_str = old_df['Texts'].str
for phrase in df.index[:nth_most_frequent]:
    old_df['Text_contains_ngrams'] |= Texts_str.contains(phrase)

Then you can use the mask to filter your df

old_df[old_df.Text_contains_ngrams]
user_55798710
Related