Creating n-grams for rows in a dataframe with counts in Pandas?

user_12730406 0
Viewed 580

I have the following dataframe:

data = {'words': ['must watch good acting',
    'average movie bad acting',
    'good movie good acting',
    'pathetic avoid',
    'avoid'], 'dates': ['2020-01-01','2020-01-01','2020-01-02','2020-01-02', '2020-01-03']}
  
df = pd.DataFrame.from_dict(data)

I would like to create bi-grams of the dataframe along with the counts for each of the dates in the dataframe that they occur. bi-grams are 2 words.

So i would like to get the end result table like this:

bi-gram count date
must watch 1 '2020-01-01'
watch good 1 '2020-01-01'
good acting 1 '2020-01-01'
average movie 1 '2020-01-01'
movie bad 1 '2020-01-01'
bad acting 1 '2020-01-01'
good movie 1 '2020-01-02'
movie good 1 '2020-01-02'
good acting 1 '2020-01-02'
pathetic avoid 1 '2020-01-02'

As you can see from the above, that the bi-gram good acting has a count of 1 but is repeated twice, once for each occurrence.

Alternatively is there a way of producing the total sum instead? So that we have the value occurring as 2 and only one rows appears, maybe with two dates nested - although this might be hard to plot?

The code that i have for producing an n-gram is the following:

from nltk import ngrams

sentence = 'test sentence'
# set n =2 for bi grams
n = 2

bigrams = ngrams(sentence.split(), n)

print(list(bigrams))

The above code gives me the bigrams, but I am stuck on how to transform the data to attain the counts of the occurrences of these bigrams, and also with the dates too.

Any help is appreciated.

pandas dataframe python
1 Answers

You can compute your ngrams, the use str.len to get the count, explode into multiple rows, and finally drop the rows with empty ngrams.

I provided an example with n=3:

n = 3
df['ngrams'] = df['words'].str.split().apply(lambda x: list(map(' '.join, ngrams(x, n=n))))
(df.assign(count=df['ngrams'].str.len())
   .explode('ngrams')
   .query('count > 0')
)

output:

                      words       dates             ngrams  count
0    must watch good acting  2020-01-01    must watch good      2
0    must watch good acting  2020-01-01  watch good acting      2
1  average movie bad acting  2020-01-01  average movie bad      2
1  average movie bad acting  2020-01-01   movie bad acting      2
2    good movie good acting  2020-01-02    good movie good      2
2    good movie good acting  2020-01-02  movie good acting      2
user_163434640
Related