Is there a way to exclude specific substrings when using str.contains?

user_6201147 0
Viewed 893

Background

I am analyzing a Pandas DataFrame that includes the title of different food recipes. My goal is to create categories for the recipe titles based on a keyword in the title.

For instance: 'Spicy Noodle Soup' would be in the Soup category, 'Sour Cream Apple Pie' would be in the Pie category.

I create DataFrames for each of these categories by using the following script

df[df['title'].str.contains('Soup', na=False, case=False)]
df[df['title'].str.contains('Pie', na=False, case=False)]

Problem

I want to exclude specific recipes that have the keyword within the string. The first example that came to mind was excluding 'pancake' from the Cake category. Although many would argue that pancake is a cake, I am not for the purpose of this excercise :) .

A non-optimal solution would be to use ' cake' as the substring, however this would exclude shortcakes, which I would like to include

Question

Is there way to exclude specific substrings when using the str.contains() method? Based on the documentation it doesn't seem like theres a built in way (I could be wrong)

pandas dataframe python
2 Answers

One thing I can think of is to replace that specific string with ''

exclude_words = ['pancake', 'cakefake']

df[df['title'].replace(exclude_words,'', regex=True)
              .str.contains('cake', case=False)
  ]

This approach will work better when you have a list of words that you want to exclude as illustrated above since you don't need to control where the cake appears relatively in your words.

Or use a negative look behind if you only have one 'pancake' word for simpler syntax:

df[df['title'].str.contains('(?<!pan)cake')]

Test data:

df = pd.DataFrame({'title':['cheesecake', 'pancake','no cake']})

Output:

        title
0  cheesecake
2     no cake
user_42384080

You can also include a condition to exclude values. The implementation would be something like this. It is a bit costly implementation though.

import pandas as pd
raw_data = {'name': ['Willard Morris', 'Al Jennings', 'Chris Cook'],
'age': [20, 19, 18],
'favorite_food': ['Cake', 'Pancake', 'Ice Cream']}
df = pd.DataFrame(raw_data)

new_df = df[df['favorite_food'].str.contains('cake', na=False, case=False)
            & ~df['favorite_food'].isin(['Pancake'])]

print ('raw-data df')
print (df)

print ('\nfiltered df for cake')
print (new_df)

The output of this will be:

raw-data df
             name  age favorite_food
0  Willard Morris   20          Cake
1     Al Jennings   19       Pancake
2      Chris Cook   18     Ice Cream

filtered df for cake
             name  age favorite_food
0  Willard Morris   20          Cake
user_138739800
Related