How to filter elements within a list and generate frequency table

user_10829044 0
Viewed 72

I have a dataframe like as shown below

df = pd.DataFrame({'text': ["Hi how","I am fine","Ila say Hi"],
                   'tokens':[['Hi','how'],['I','am','fine'],['Ila','say','Hi']],
                    'labels':[['A','B'],['C','B','A'],['D','B','A']]})

I would like to do two things

a) count of each labels

b) count tokens under each label

I was trying something like below

flattened = [] 
op = itertools.zip_longest(df['tokens'],df['labels'])
for i in op:
    for la in df['labels']: 
        if la == 'A' : 
        flattened.append(val)

But this is incorrect and going no-where

I expect my output to have two dataframes/tables like as shown below

enter image description here

pandas dataframe nlp python-3.x python
2 Answers

We can use value_counts after flattening the columns tokens and labels using hstack

t = np.hstack(df['tokens'])
l = np.hstack(df['labels'])

Count of each label

pd.value_counts(l)

B    3
A    3
D    1
C    1
dtype: int64

Count tokens under each label

pd.DataFrame(zip(l, t)).value_counts()

0  1   
A  Hi      2
   fine    1
B  am      1
   how     1
   say     1
C  I       1
D  Ila     1
dtype: int64
user_128331660

You can also use Pandas's explode method:

>>> df2 = df[['labels', 'tokens']].apply(pd.Series.explode)
>>> df2.labels.value_counts()

B    3
A    3
D    1
C    1
Name: labels, dtype: int64

>>> df2[['labels', 'tokens']].value_counts()
labels  tokens
A       Hi        2
        fine      1
B       am        1
        how       1
        say       1
C       I         1
D       Ila       1
dtype: int64
user_158730430
Related