Python: Tag keywords and create new columns of tags with 1's and 0's

Admin 1
Viewed 206

I have the below code to iterate through sentences of a column, tag keywords in sentences, and create new columns of those tags consisting of 1's and 0's. If a keyword exists, it is automatically tagged and given a 1 in a new column named after the tag. If it does not exist but another keyword exists, it is given a 0. If the sentence does not have any keywords whatsoever, the entire row will be removed.

The code below is somewhat okay but it still misses keywords and it tags and outputs 1's and 0's on partial words and blank cells (rows with no sentences). I'm not sure what is missing? How do I ensure that it does not miss keywords and does not tag partial words and blank sentences?

pattern = '|'.join(dict_list)
tags_id = (df['description_summary']
   .str.extractall(f'({pattern})')[0]
   .map(keyword_dict)
   .reset_index(name='col')
   .assign(value=1)
   .pivot_table(index=[df['issue.id'], df['description_summary']], columns='col', values='value', fill_value=0))

Here is basically the data I'm working with in an excel file:

    issue.id  description_summary

0   753       Long sentence with keywords ball and hot
1   937       Long sentence with keywords cold, stick, and glove
2   
3   598       Long sentence with NO keywords
4   574       Long sentence with keywords very cold and cold

Here is the current (wrong) output:

    issue.id  description_summary                                     Toy     Temperature 

0    753       Long sentence with keywords ball and hot                1       1
1    937       Long sentence with keywords cold, stick, and glove      1       1
2                                                                      1       0
3    598       Long sentence with NO keywords but outputs 1s and 0s    0       1
4    574       Long sentence with keywords very cold and cold          1       1

Here is the output I want:

    issue.id  description_summary                                     Toy     Temperature    

0    753       Long sentence with keywords ball and hot                1       1
1    937       Long sentence with keywords cold, stick, and glove      1       1
4    574       Long sentence with keywords very cold and cold          0       1

Here is the dictionary of keywords and tags ('keywords': 'tags'):

dict_list = {'Hot': 'Temperature',
 'Cold': 'Temperature',
 'Very cold': 'Temperature',
 'Ball': 'Toy',
 'Glove': 'Toy',
 'Stick': 'Toy'
 }

How do I ensure that it does not miss keywords and does not tag partial words and blank sentences?

pandas nlp tags keyword python
2 Answers

I think your first issue is with map. If I reconstitute roughly what you’re doing until there:

>>> pattern = '|'.join(dict_list.keys())
>>> matches = df['description_summary'].str.extractall(f"({pattern})", flags=re.IGNORECASE)[0]
>>> matches
   match
0  0             ball
   1              hot
1  0             cold
   1            stick
   2            glove
4  0        very cold
   1             cold
Name: 0, dtype: object
>>> matches.map(dict_list)
   match
0  0        NaN
   1        NaN
1  0        NaN
   1        NaN
   2        NaN
4  0        NaN
   1        NaN
Name: 0, dtype: object

However enforcing case insensitiveness we get a better result:

>>> matches.str.lower().map({kw.lower():tag for kw, tag in dict_list.items()})
   match
0  0                Toy
   1        Temperature
1  0        Temperature
   1                Toy
   2                Toy
4  0        Temperature
   1        Temperature
Name: 0, dtype: object

The second issue seems to be pivot_table which assigns the wrong lines to matches because df and matches do not have the same shape. We can instead pivot using the first level of index, and then use that to join with df:

>>> tags = matches.str.lower().map({kw.lower():tag for kw, tag in dict_list.items()})
>>> tags = tags.rename_axis(['line', 'match']).reset_index(name='tag').assign(value=1)
>>> tags.pivot_table(index='line', columns='tag', values='value', fill_value=0).join(df[['issue.id', 'description_summary']])
      Temperature  Toy  issue.id                                description_summary
line                                                                               
0               1    1     753.0           Long sentence with keywords ball and hot
1               1    1     937.0  Long sentence with keywords cold, stick, and g...
4               1    0     574.0     Long sentence with keywords very cold and cold
user_13873460

Input data:

>>> df
  issue.id                                description_summary
0      753           Long sentence with keywords ball and hot
1      937  Long sentence with keywords cold, stick, and g...
2     <NA>                                               <NA>
3      598                     Long sentence with NO keywords
4      574     Long sentence with keywords very cold and cold

>>> mapping
{'Hot': 'Temperature',
 'Cold': 'Temperature',
 'Very cold': 'Temperature',
 'Ball': 'Toy',
 'Glove': 'Toy',
 'Stick': 'Toy'}

>>> words  # words = fr"({'|'.join(mapping.keys())})".lower()
'(hot|cold|very cold|ball|glove|stick)'

I will write some explanation later (but you can test line by line)

out = df['description_summary'].str.lower().str.findall(words) \
                               .explode().str.capitalize() \
                               .replace(dict_list) \
                               .pipe(lambda x: x.loc[x.notna()]) \
                               .str.get_dummies() \
                               .groupby(level=0) \
                               .any().astype(int)

Output result

>>> df.merge(out, left_index=True, right_index=True)
  issue.id                                description_summary  Temperature  Toy
0      753           Long sentence with keywords ball and hot            1    1
1      937  Long sentence with keywords cold, stick, and g...            1    1
4      574     Long sentence with keywords very cold and cold            1    0
user_152399510
Related