Get duplicate line and the rest of the file [ Big file 50G]

user_6802333 0
Viewed 138

i have a big file 50G and i want to get the duplicate line and rest of file i use two command to get the result and that Take a long time.

sort file.tsv | uniq -d > duplicateList.tsv 
sort file.tsv | uniq -u > clean_List.tsv

As you can see the process repeated twice , i want to make only one commande and return the both result without using duplicate commande

note i can use linux commande or Python script

sed python
3 Answers

This might work for you (GNU sed), presumes the file is already sorted:

sed -Ee 'N;/^(.*)\n\1$/!{P;D};:a;$w duplicatesFile' \
     -e '$d;N;/(\n.*)\1$/ba;h;s/.*\n/\n/;x;s/(.*)\n.*/\1/w duplicatesFile' \
     -e 'x;D' file > nonduplicatesFile

In overview: the duplicates are written out to the duplicatesFile and the remainder to stdout which is redirected to nonduplicatesFile.

Initially a 2 line buffer is created and patterned matched for duplicate lines. If not, the first line is printed to stdout and then deleted and repeated until a duplicate line occurs.

For duplicate lines the edge case of end-of-file is handled first, where all the remaining lines are output to the duplicatesFile and processing is halted.

For the other case, the pattern space is copied to the hold space and then split into duplicates and non-duplicate. The duplicates written to then duplicatesFile and the non-duplicate, prepended with a newline and then the newline deleted using the D command which causes the sed commands to be rerun less the implicit fetching of the next line from file.

N.B. Sed is never the fastest solution perhaps in this case a dedicated piece of code might provide the required speed.

user_9674920

Hashing is by far the fastest. Here's how in python:

name = '/my/big/file'
lines = dict()
dups  = dict()
with open( name ) as f:
    line = line.rstrip()
    for line in f:
        if line in lines:
            dups[ line ] = True
        else:
            lines[ line ] = True
print( 'Duplicate(s)' )
for line in dups:
    print( line )
print( 'Unique(s)' )
for line in lines:
    if line not in dups:
        print( line )

Python dictionaries are implemented as a hashed collection.

user_5379590

I think you are looking for this syntax:

sort file.tsv > >(uniq -d > duplicateList.tsv) > >(uniq -u > clean_List.tsv)

It will send the stdout to both commands, so sorting only happens once, and the other two in parallel.

user_52863930
Related