Remove duplicate lines without sorting

user_916332 0
Viewed 74583

I have a utility script in Python:

#!/usr/bin/env python
import sys
unique_lines = []
duplicate_lines = []
for line in sys.stdin:
  if line in unique_lines:
    duplicate_lines.append(line)
  else:
    unique_lines.append(line)
    sys.stdout.write(line)
# optionally do something with duplicate_lines

This simple functionality (uniq without needing to sort first, stable ordering) must be available as a simple UNIX utility, mustn't it? Maybe a combination of filters in a pipe?

Reason for asking: needing this functionality on a system on which I cannot execute Python from anywhere.

uniq filter shell scripting
8 Answers

Now you can check out this small tool written in Rust: uq.

It performs uniqueness filtering without having to sort the input first, therefore can apply on continuous stream.

There are two advantages of this tool over the top-voted awk solution and other shell-based solutions:

  1. uq remembers the occurence of lines using their hash values, so it doesn't use as much memory use when the lines are long.
  2. uq can keep the memory usage constant by setting a limit on the number of entries to store (when the limit is reached, there is a flag to control either to override or to die), while the awk solution could run into OOM when there are too many lines.
user_12328320
user_49283880
Related