Count the occurrence words and tags in each row of dataframe?

user_8921989 0
Viewed 356

I have a code that performs the Part of speech tagging using nltk and next, I am counting how many tags occurred in each row.

Example: If I have a string input as:

A = [('car', 'NN'), ('bike', 'NN')]

I tried to write my own for loop as below as well:

def count_tags(input_col):
    tag_count = {}
    for word, tag in input_col:
        if tag in tag_count:
            tag_count[tag] += 1
        else:
            tag_count[tag] = 1
    return(tag_count)

print('tag count is: ', count_tags(hi))

which output the same as:

tag count is:  {'NN': 2}

Question:

I have a CSV file that contains strings in each row as like A example above:

print(df['tags'])

0    [('car', 'NN'), ('bike', 'NN')]
1    [('TV', 'NN'), ('access', 'JJ')

and when I load with panda and then tried to count the tags:

df['tag_count'] = df['tags'].map(count_tags)

I am getting the following error:

ValueError: not enough values to unpack (expected 2, got 1)

My expected output is:

ID  tags                              count
2   [('car', 'NN'), ('bike', 'NN')]   {'NN': 2}
4   [('TV', 'NN'), ('access', 'JJ')]  {'NN': 1, 'JJ': 1}
python
1 Answers

Solution

# Using the dataframe df from the Dummy Data section
df2 = (df.tags.explode()
        .explode().groupby(level=0).value_counts()
        .to_frame().rename(columns={'tags': 'count'})
        .reset_index().rename(columns={'tags': 'tag'}) 
)
# Show all tags count
header = lambda s: f'\n{s}\n' + ''.join(['=']*len(s))
print(header('All Tags Count'))
print(df2)

target_tags = ['NN', 'JJ']
# Show target_tags count
result = df2.loc[df3.tag.str.contains('|'.join(target_tags))]
#print("\nTarget Tags Count\n")
print(header('Target Tags Count'))
print(result)

Output:

All Tags Count
==============
   ID   tag  count
0   2    NN      2
1   2  bike      1
2   2   car      1
3   4    JJ      1
4   4    NN      1
5   4    TV      1
6   4   pen      1

Target Tags Count
=================
   ID tag  count
0   2  NN      2
3   4  JJ      1
4   4  NN      1

Dummy Data

import pandas as pd
df = pd.DataFrame([[2, [('car', 'NN'), ('bike', 'NN')]],
                   [4, [('TV', 'NN'), ('pen', 'JJ')]]], 
                  columns = ['ID', 'tags'])
df.set_index(["ID"], drop=False, inplace=True) # important prerequisite for the solution
print(df)

## Output: df
#
#     ID                     tags
# ID                             
# 2    2  [(car, NN), (bike, NN)]
# 4    4    [(TV, NN), (pen, JJ)]
user_84748940
Related