Remove reverse matches in bash

user_8908056 0
Viewed 264

I have a file, list.txt that contains:

234
243
324
342
423
432

How can I find if reverse patterns (i.e., 432 is the reverse pattern of 234) exists and remove the reverse pattern? I have attempted

while IFS= read -r line; do
  reverse=$(echo $line|rev)
  if grep -q $reverse list.txt; then
    sed -i "s/$reverse//g" list.txt
  else :
  fi
done < list.txt

but this removes every line from list.txt. My expected output would be

234
243
324

Is what I want to accomplish possible? My MWE is a short list, but this list can (and will) grow considerably. Thanks in advance.

awk bash
2 Answers

Original Question: Removing All Items With Reverses In The Input

Removing all strings that are inverses of any other string in the file would look like:

grep -Fvf <(rev list.txt) <list.txt >list.txt.new && mv list.txt.new list.txt

Let's break that down:

  • grep -F matches only fixed strings.
  • grep -v inverts the match, emitting things that don't match.
  • grep -f filename reads the list of patterns to look for from filename.
  • <(rev list.txt) is a process substitution that expands to a filename from which the output of rev list.txt can be read.
  • <list.txt connects list.txt to the stdin of your grep.
  • >list.txt.new connects stdout of grep to a new file; this is important, since >list.txt would overwrite your output file before its original contents could be read.

However, with your sample input, this results in completely empty output -- because every line in that sample input file has a reverse version elsewhere in that file.

Refined Question: Removing Only Inverses Not Previously Seen

Given your sample data, you don't really want to remove all data that has an inverse version somewhere else in the input file. Instead, you want to read top-to-bottom, and print only things whose inverses weren't already seen.

One way to do that would be the following:

#!/usr/bin/env bash
case $BASH_VERSION in ''|[123].*) echo "ERROR: Bash 4.0+ needed" >&2; exit 1;; esac

declare -A blacklisted=( )
while IFS= read -r orig <&3 && IFS= read -r rev <&4; do
  [[ ${blacklisted[$orig]} ]] && continue
  blacklisted[$rev]=1
  printf '%s\n' "$orig"
done 3< list.txt 4< <(rev list.txt) >list.txt.new && mv list.txt.new list.txt

BTW, note that in the real world, instead of hardcoding something like list.txt.new, you should use mktemp to create a guaranteed-unique/random name for your temporary files. This doesn't just fix concurrency issues -- it also fixes security bugs.

user_141220

Here is an awk solution:

awk 'BEGIN{FS=""} !seen[$0]{s=""; for (i=NF; i>0; i--) s=s $i; seen[s]++; print}' file


234
243
324

Explanation:

  • BEGIN{FS=""}: Set input file separator to empty string so that every character in input becomes a field in awk.
  • !seen[$0] {: if current row is not found in seen array
    • s="";: Initialize s to empty string
    • for (i=NF; i>0; i--) s=s $i: Run a reverse loop and strore reverse string in s
    • seen[s]++;: Store s in array seen
    • print: Print current row
user_5482250
Related