Use a dictionary to find/replace exact terms in a tokenized pandas series

user_9847032 0
Viewed 152

Using a dictionary, I need to find and replace terms in a pandas series according to the following criteria:

  1. dictionary values are replacing dictionary keys wherever they are found in the pandas series (e.g., for example, in 'mastersphd': 'masters phd', the replacement outcome would be 'masters phd' wherever 'mastersphd' occurs)
  2. maintain record integrity (i.e., can't use a bag of words approach because I need the unique records to remain intact.)
  3. Only exact matches should be replaced (for example, if key:value is 'rf': 'random forest', the replacement should not turn 'performance' into 'perandom forestormance'); so regex=True is causing this, obviously)

The Data: term_fixes is the dictionary, and df['job_description'] is the tokenized series of interest

term_fixes = {'rf': 'random forest',
              'mastersphd': 'masters phd',
              'curiosity': 'curious',
              'trustworthy': 'ethical',
              'realise': 'realize'}


df = pd.DataFrame(data={'job_description': [['knowledge', 'of', 'algorithm', 'like', 'rf'],
                                            ['must', 'have', 'a', 'mastersphd'],
                                            ['trustworthy', 'and', 'possess', 'curiosity'],
                                            ['we', 'realise', 'performance', 'is', 'key']]})

**Note: I have also (unsuccessfully) tried an untokenized data structure, but prefer tokenized since I have more NLP to do

df = pd.DataFrame(data={'job_description': ['knowledge of algorithm like rf',
                                            'must have a mastersphd',
                                            'must be trustworthy and possess curiosity',
                                            'we realise performance is critical']})

**Desired Outcome (note that the 'rf' in performance is not replaced by 'random forest'): df['job_description']

0    ['knowledge' 'of' 'algorithm' 'like' 'random' 'forest']
1                        ['must' 'have' 'a' 'masters' 'phd']
2          ['must' 'be' 'ethical' 'and' 'possess' 'curious']
3             ['we' 'realize' 'performance' 'is' 'critical']

I have tried a number of approaches. Fail: df['job_description'].replace(list(term_fixes.keys()), list(term_fixes.values()), regex=False, inplace=True)

Fail: df['job_description'].replace(dict(zip(list(term_fixes.keys()), list(term_fixes.values()))), regex=False, inplace=True)

Fail: df['job_description'] = df['job_description'].str.replace(term_fixes, regex=False)

Fail: df['job_description'] = df['job_description'].str.replace(str(term_fixes.keys()), str(term_fixes.values()), regex=True)

The closest I have come is:

df['job_description'] = df_jobs['job_description'].replace(term_fixes, regex=True)

However, the regex=True flags on any match (like the 'rf' and 'performance' example above). Changing the flag to regex=False fails to replace anything, unfortunately. I looked in the documentation for another argument I could use, but no luck. Note that this uses the untokenized structure.

Any help would be much appreciated. Thanks!

pandas dataframe nltk nlp python
2 Answers

You could use something like the following to do so with the untokenized data.

for k in term_fixes:
    df['job_description'] = (df['job_description'].str.replace(r'(^|(?<= )){}((?= )|$)'.format(k), term_fixes[k]))

print(df)
                             job_description
0  knowledge of algorithm like random forest
1                    must have a masters phd
2        must be ethical and possess curious
3         we realize performance is critical
user_147222970

Using the "tokenized" version of your df.

df['job_description'] = df['job_description'].explode().replace(term_fixes).groupby(level=-1).agg(list)

# explode to get single terms per "cell"
# replace to replace the terms in "term_fixes"
# groupby to reverse the previous explode and return to a column of lists

                                   job_description
0  [knowledge, of, algorithm, like, random forest]
1                     [must, have, a, masters phd]
2                 [ethical, and, possess, curious]
3              [we, realize, performance, is, key]

If you need the new terms also split on white space, then you can add another intermediate step .str.split().explode() before the final groupby

df['job_description'] = df['job_description'].explode().replace(term_fixes).str.split().explode().groupby(level=-1).agg(list)

                                    job_description
0  [knowledge, of, algorithm, like, random, forest] # random forest is now split
1                     [must, have, a, masters, phd] # masters phd is now split
2                  [ethical, and, possess, curious]
3               [we, realize, performance, is, key]
user_167462530
Related