Remove duplicates using pandas & Python

Viewed 1547

I want to remove duplicates based on my first column, Lets assume this to be 'id'. the value that I need to be removed is the records with the least complete data.

For example I have 4 records shown in the screenshot below enter image description here

and I want to remove duplicated based on my column id. The duplicate I would like to be removed however depends on the values in my other columns. For example lname cant be a number. It cant be NA or blank either. So for this example I want to keep the middle row and remove all other duplicates.

How do I implement a smart filter that removes duplicates by keeping the records that looks the most clean ( based on there being cleaner values in other fields).

I was looking into the panda library in python. Any direction would be appreciated.

data = pd.read_csv('x.csv');
data = (data.drop_duplicates(['id'], keep ='last'));
2 Answers
Related