Get most frequent words in list for each row

user_11423076 0
Viewed 81

I have a dataframe df with 50,000+ rows:

>>> df
                 message                          words  wordCount  uniqueWordCount
0             my name is                 [my, name, is]          3                3
1  happy birthday to you     [happy, birthday, to, you]          4                4
2         la la la la la           [la, la, la, la, la]          5                1
3 you are you that is it  [you, are, you, that, is, it]          6                5
...

I want to create a new column with the 3 most frequent words in the message. What I have so far works, but takes quite some time.

>>> df["mostFrequent"] = df["message"].apply(
      lambda x: sorted(
        textblob.TextBlob(x).word_counts, key=textblob.TextBlob(x).word_counts.get, reverse=True)[:3])

>>> df["mostFrequent"]
0 [my, name, is]
1 [happy, birthday, to]
2 [la]
3 [you, are, that]
...

Is there a more efficient way of doing this?

pandas dataframe python-3.x python
1 Answers

Use custom lambda function with collections.Counter:

from collections import Counter
f = lambda x: [word for word, word_count in Counter(x).most_common(3)]
df["mostFrequent"] = df["words"].apply(f)
print (df)
                  message                          words  wordCount  \
0              my name is                 [my, name, is]          3   
1   happy birthday to you     [happy, birthday, to, you]          4   
2          la la la la la           [la, la, la, la, la]          5   
3  you are you that is it  [you, are, you, that, is, it]          6   

   uniqueWordCount           mostFrequent  
0                3         [my, name, is]  
1                4  [happy, birthday, to]  
2                1                   [la]  
3                5       [you, are, that]
user_29010020
Related