Pandas: How to drop column values that are duplicates but keep certain row values

userxGd51 0
Viewed 36

I have a Pandas dataframe that have duplicate names but with different values, and I want to remove the duplicate names but keep the rows. A snippet of my dataframe looks like this: enter image description here

And my desired output would look like this:

enter image description here

I've tried using the builtin pandas function .drop_duplicates(), but I end up deleting all duplicates and their respective rows. My current code looks like this:

df = pd.read_csv("merged_db.csv", encoding = "unicode_escape", chunksize=50000)
df = pd.concat(df, ignore_index=True)
df2 = df.drop_duplicates(subset=['auth_given_name', 'auth_surname'])

and this is output I am currently getting:

enter image description here

Basically, I want to return all the values of the coauthor but remove all duplicate data of the original author. My question is what is the best way to achieve the output that I want. I tried using the subset parameter but I don't believe I'm using it correctly.I also found a similar post, but I couldn't really apply it to python. Thank you for your time!

dataframe pandas python
1 Answers

You may consider this code

df = pd.read_csv("merged_db.csv", encoding = "unicode_escape", chunksize=50000)
first_author = df.columns[:24]
df.loc[df.duplicated(first_author), first_author] = np.empty(len(first_author))
print(df)
userYE4D0
Related