Issue removing multiple duplicate lines from a text file

user_13167041 0
Viewed 57

I am trying to remove duplicate lines from a text file and keep facing issues... The output file keeps putting the first two accounts on the same line. Each account should have a different line... Does anyone know why this is happening and how to fix it?

with open('accounts.txt', 'r') as f:
    unique_lines = set(f.readlines())
with open('accounts_No_Dup.txt', 'w') as f:
    f.writelines(unique_lines)

accounts.txt:

@account1
@account2
@account3
@account4
@account5
@account6
@account7
@account5
@account8
@account4

accounts_No_Dup.txt:

@account4@account3
@account4
@account8
@account5
@account7
@account1
@account2
@account6

print(unique_lines)

{'@account4', '@account7\n', '@account3\n', '@account6\n', '@account5\n', '@account8\n', '@account4\n', '@account2\n', '@account1\n'}
python
3 Answers

The last line in your file is missing a newline (technically a violation of POSIX standards for text files, but so common you have to account for it), so "@account4\n" earlier on is interpreted as unique relative to "@account4" at the end. I'd suggest unconditionally stripping newlines, and adding them back when writing:

with open('accounts.txt', 'r') as f:
    unique_lines = {line.rstrip("\r\n") for line in f}  # Remove newlines for consistent deduplication
with open('accounts_No_Dup.txt', 'w') as f:
    f.writelines(f'{line}\n' for line in unique_lines)  # Add newlines back

By the by, on modern Python (CPython/PyPy 3.6+, 3.7+ for any interpreter), you can preserve order of first appearance by using a dict rather than a set. Just change the read from the file to:

    unique_lines = {line.rstrip("\r\n"): None for line in f}

and you'll see each line the first time it appears, in that order, with subsequent duplicates being ignored.

user_3646960

Your problem is that set changes the order of your lines and your last element doesn't end with \n as you don't have an empty line at the end of your file.

Just add the separator or don't use set.

with open('accounts.txt', 'r') as f:
    unique_lines = set()
    for line in f.readlines():
        if not line.endswith('\n'):
            line += '\n'
        unique_lines.add(line)


with open('accounts_No_Dup.txt', 'w') as f:
    f.writelines(unique_lines)
user_58334290

You can easily do it using unique keyword

The code is as below

import pandas as pd

data = pd.read_csv('d:\\test.txt', sep="/n", header=None)
df =  pd.DataFrame(data[0].unique())

with open('d:\\testnew.txt', 'a') as f:
    f.write(df.to_string(header = False, index = False)))

Results: Test file to read has data

enter image description here

The result is it removed the duplicate lines

enter image description here

user_109436050
Related