Pandas - Create multiple filters and apply to dataframe

user_11227857 0
Viewed 99

I have created multiple filters for a Dataframe:

filt1 = ~df["message"].str.contains("<Media omitted>", na=False),
filt2 = ~df["message"].str.contains("http://", na=False),
filt3 = ~df["message"].str.contains("Dropped pin", na=False),

I can filter the dataframe using:

df[filt1 & filt2 & filt3]

But as I add more filters this seems like a stupid way to filter. How do I apply multiple filters to a dataframe?

I tried adding each filter to a list doing df[filterlist] and df[*filterlist] but these do not work.

pandas python
2 Answers

You can use np.logical_and.reduce:

filterlist = [filt1, filt2, filt3]

df[np.logical_and.reduce(filterlist)]

Or concat with DataFrame.all for test all Trues per rows:

df[pd.concat(filterlist, axis=1).all(axis=1)]

If possible use | for regex or:

filt = ~df["message"].str.contains("<Media omitted>|http://|Dropped pin", na=False)
user_29010020

You can use join to merge list for regex

df[~df.message.str.contains('|'.join(filterlist))]
user_60756990
Related