Trying to remove repetitions in a paragraph using BASH

user_2835563 0
Viewed 62

HI I am writing a simple BASH to remove the consecutive duplicate occurrence of any word in a paragraph separated by single space and redirecting output to stdout, here is how far I got.

file1

**double double toil and trouble 
fire burn and cauldron bubble bubble 
tomorrow and tomorrow and tomorrow 
creeps in this this petty pace from day toto day**

echo `<file1` | sed -e 's/\b\([a-z ]\+\)\1/\1/g' | cat > file2

This is outputting in a single line as below.

double toil and trouble fire burn and cauldron bubble tomorrow and tomorrow creeps in this petty pace from day to day

and missing something as it is not correctly removing occurrences.

shell linux bash regex
1 Answers

You can use

sed 's/\b\([a-z]\+\)\s\1\b/\1/g' file > file1
sed 's/\b\([a-z]\+\)[[:space:]]\1\b/\1/g' file > file1

See the online demo. The regex matches

  • \b - a word boundary
  • \([a-z]\+\) - Group 1: any one or more lowercase letters
  • [[:space:]] / \s - a whitespace
  • \1 - same value as in Group 1
  • \b - a word boundary.
user_38329700
Related