Applying filtering to a pandas dataframe

user_7700802 0
Viewed 48

I am trying to segment the dataframe where I only have a dataframe that contains certain words inside one of its columns and does not contain others.

For example

d = {'resolution' : ['replaced scanner', 'replaced the scanner for the user with a properly working one from the cage replaced the wire on the damaged one and stored it for later use','the scanner has been replaced and the station is working well now', 'replaced scanner screen']}
df_sample = pd.DataFrame(data=d)

keywords = ['replace', 'replaced', 'change', 'changed', 'swapped', 'deployed', 'scanner']
filtered_words = ['screen']
df_filtered = df_sample[(df_sample['resolution'].isin(keywords)) & (~df_sample['resolution'].isin(filtered_words))]
df_filtered.head()

I would expect to get

d = {'resolution' : ['replaced scanner', 'replaced the scanner for the user with a properly working one from the cage replaced the wire on the damaged one and stored it for later use','the scanner has been replaced and the station is working well now']}
df_filtered = pd.DataFrame(data=d)

But instead I just get an empty dataframe. Any suggestions are appreciated.

pandas python
2 Answers

Another way is to chain and mask str.contains(keywords) & ~str.contains(filtered_words) . Where ~ denotes doesnot

df_sample[df_sample['resolution'].str.contains("|".join(keywords))&~df_sample['resolution'].str.contains("|".join(filtered_words))]

                             

                            resolution
0                                   replaced scanner
1  replaced the scanner for the user with a prope...
2  the scanner has been replaced and the station ...
user_89869750

You can use .apply() with any():

m_keywords = df_sample["resolution"].apply(
    lambda x: any(k in x for k in keywords)
)
m_filtered = df_sample["resolution"].apply(
    lambda x: any(k in x for k in filtered_words)
)

print(df_sample[m_keywords & ~m_filtered])

Prints:

                                          resolution
0                                   replaced scanner
1  replaced the scanner for the user with a prope...
2  the scanner has been replaced and the station ...
user_100359850
Related