Drop Duplicates from Pandas DF with Conditions

user_15952844 0
Viewed 126

I have a pandas df that looks like the below:

Current df:

Case Task Text_Present
123 Email Yes
123 Outreach No
456 Session No
456 Email No

I would like to drop duplicate cases, but based on a condition of the Text_Present field.

If there's a 'Yes' present, keep the record where 'Yes' is present. If there's no 'Yes' present (ex: Case 456), then keep one of the records for 456 - does not matter which one.

I'm basically looking to understand the % Yes to % No, but the math is off due to the fact that the cases are duplicated as a result of the Description field - since I want to count any case where there was a 'Yes' in any Task as just a 'Yes' overall for that case.

My goal output is:

Case Task Text_Present
123 Email Yes
456 Session No

Ultimately, I want to compare % Yes to % No in a simple barchart, but the duplication is resulting in inaccurate percentages.

As far as I can tell, df.drop only offers first and last.

Thanks All!

drop pandas dataframe duplicates python
1 Answers

You can sort the values with respect to Case and Text_Present. Then you can drop the duplicates over Case column and keep the last ones. Since "Yes" comes after than "No" alphabetically, it will reside in last positions and will be kept:

>>> df.sort_values(["Case", "Text_Present"]).drop_duplicates("Case", keep="last")

   Case   Task Text_Present
0   123  Email          Yes
3   456  Email           No
user_93321870
Related