How to remove duplicate lines without creating empty lines in CSV file?

user_16601417 0
Viewed 236

CSV file:

a,b
a,c
a,d
a,b
a,a

Code widely recommended for removing duplicates:

import fileinput
seen = set()
for line in fileinput.FileInput('1.csv', inplace=1):
    if line in seen: continue

    seen.add(line)
    print(line)

Result obtained:

a,b

a,c

a,d

a,a

expected result:

a,b
a,c
a,d
a,a

What should I do to not create these lines during the process?

csv python
2 Answers

The print function adds a new line at the end of each line, to change the behavior add the following argument like this.

import fileinput
seen = set()
for line in fileinput.FileInput('1.csv', inplace=1):
    if line in seen: continue

    seen.add(line)
    print(line,end='')

There are many other ways and other libraries you can use to achieve this, this post https://www.py4u.net/discuss/16763 covers the other methods quite well, you can go through all of them and see which one works the best for you.

user_116937680

First why this happens is because you are reading a line which ends with \n and the print() function is also adding another \n. If you wish to print the distinct lines only, you can print them using print(line, end='').

However, if you wish to remove duplicate rows, I recommend using pandas like:

import pandas as pd
df = pd.read_csv('1.csv')

# Removing duplicate rows: 
df = df.drop_duplicates()

# Saving after removing duplicates
df.to_csv('1_clean.csv', index=False)
user_32796030
Related