Get word counts in strings of words in a list using python

user_156957 0
Viewed 106

Starting with a pandas data frame where the first column is made up of comment strings with the other columns being features of single words. For each row I would like to get a count of how many times each word appears in that row's comments cell. I have the list of words (feature columns) as a list called, "wordList" and am trying something like this but having trouble getting it working and the counts back into the data frame:

def word_count(comments):
    for word in wordList:
        return comment.count(word)

df.comments.apply(word_count)

What I have:

comments        |  hello  |  this  |   is   |  the  |  comments  |  blah  |
--------------------------------------------------------------------------
this is the 1st |         |        |        |       |            |    
comments here   |         |        |        |       |            |
--------------------------------------------------------------------------
the 2nd comment |         |        |        |       |            |    
is is here this |         |        |        |       |            |

What I want:

comments        |  hello  |  this  |   is   |  the  |  comments  |  blah  |
--------------------------------------------------------------------------
this is the 1st |    0    |    1   |   2    |   1   |     1      |    0
comments is here|         |        |        |       |            |
--------------------------------------------------------------------------
the 2nd comment |    0    |    1   |   2    |   2   |     0      |    0
is is here the  |         |        |        |       |            |
pandas word count python
2 Answers

Convert your coment column into a list and explode.

Apply get dummies. That will tabulate frequency of occurrence

Reindex with the list of words you wish to check

Aggregate the frequency and join to the df.coments column

Code below:

g=pd.get_dummies(pd.Series(df1.coments.str.split('\s').explode())).reindex(columns=['hello', 'this','is','the','comments','blah']).fillna(0).astype(int)

pd.DataFrame(df1.iloc[:,0]).join(g.groupby(level=0).sum(0))




     coments                         hello  this  is  the  comments  blah
0    this is the 1st comments here      0     1   1    1         1     0
1  the 2nd comment is is here this      0     1   2    1         0     0
user_89869750

You can use str.extract to extract (only) the words in your word list, then use value_counts:

pattern = '|'.join(word_list)
(df.comments.str.extractall(rf'\b({pattern})\b')[0]
   .groupby(level=0).value_counts()
   .unstack(fill_value=0)
   .reindex(word_list, axis=1, fill_value=0)
)

Output (note that this also has a column named comments as in the original dataframe)

0  hello  this  is  the  comments  blah
0      0     1   1    1         1     0
1      0     1   2    1         0     0
user_42384080
Related