pandas compare header with rows and drop duplicate rows

user_13595465 0
Viewed 591

I have a dataframe where i have repeated column headers in rows. I need to drop only duplicated header name and not duplicated values. Please note header is None in this case

Input:

0. Gender age occu name 
1. F      21   A    B    
2. Gender age occu name
3. M      22   C    D
4. Gender age occu name
5. M      23   F    E
6. M      23   F    E

Required output:

0. Gender age occu name 
1. F      21   A    B    
2. M      22   C    D
3. M      23   F    E
5. M      23   F    E

For this i'm trying:

for i in range(0,len(a)-1):
print(a.iloc[i])
if(a.loc[0] == a.loc[i+1]).all() == True:
    print('yes')
    a.drop(i)

Please note i can't take the row name(0th row directly as it may changes and that's why i have to compare all the other rows with 0th row using loop and have to drop that row where ever there is a match). Consider 0th row as primary row.

pandas-groupby pandas python-3.x numpy python
1 Answers

If need all rows if not match first column by first column name use boolean indexing with DataFrame.iloc:

print (df)
   Gender  age  occu  name
0       F   21     A     B
1  Gender  age  occu  name
2       M   22     C     D
3  Gender  age  occu  name
4       M   23     F     E
5       M   23     F     E

df1 = df[df.iloc[:, 0] != df.columns[0]]
#alternative
#df1 = df[df.iloc[:, 0].ne(df.columns[0])]
print (df1)
  Gender age occu name
0      F  21    A    B
2      M  22    C    D
4      M  23    F    E
5      M  23    F    E
user_29010020
Related