Find the frequency of words of a dataframe based on another dataframe

John Angelopoulos 0
Viewed 26

I have a general dataframe with the following form

df = 
index   text
0   I really dont come across how I actually am an...
1   music has become the only way I am staying san...
2   adults are contradicting
3   exo are breathing 553 miles away from me. they...
4   I am missing people that I met when I was hospit...
... ... ...
3365023 3365023 I don't want to hear it. I really really fuckin...

and I have another dataframe with the following form

df_frequency = 
Unique Words    Label
     i            2
     to           2
     the          2
    .....        ...
    @gaiaxys      1

and I want to add a new column in the df_frequency dataframe called 'frequency' which contains the frequency of each word in the dataframe df.

dataframe pandas nlp python
1 Answers

try:

#1. create new dataframe with words frequency from the column df['text']
a = [i.split() for i in df['text'].tolist()]
df2 = pd.Series([i.strip() for x in a for i in x]).value_counts().reset_index(name='frequency').rename(columns={'index':'unique_words_from_df'})

#df2
    unique_words_from_df    frequency
0   i                       5
1   am                      2
2   are                     2
3   come                    1
4   exo                     1
5   was                     1
#...

#2. merge df_frequency with df2
df_frequency.merge(df2, right_on='unique_words_from_df', left_on='Unique Words', how='left').drop(columns='unique_words_from_df')

Unique Words    Label   frequency
i               2       5.0
to              2       NaN
the             2       1.0
khaled koubaa0
Related