How do I remove words in a file from a list of words in another .txt file using bash?

Viewed 634

I'm trying to remove words in a list (a .txt file) using sed, but it is not working correctly. The output file removes the words it should remove and also parts of words it shouldn't remove.

This is the code that I've tried:

sed -E 's/('"$(tr '\n' '|' < listOfWords.txt )"')//gI' file.txt > output.txt

Example of input:

I would love to try or hear the sample audio your app can produce. I do not want to purchase, because I've purchased so many apps that say they do something and do not deliver.  

Can you please add audio samples with text you've converted? I'd love to see the end results.

Thanks!

Expected output:

would love try hear sample audio app can produce. do want purchase, because ve purchased many apps say do something do deliver.  

Can please add audio samples text ve converted? d love see end results.

list of words sample:

...
I
to
or
the
your
not
so
that
they
and
you
with
Thanks
...
4 Answers

Using \< and \> in your sed expression prevents items that don't start and end at word boundaries from being removed. Be aware that this is a GNUism -- a baseline POSIX-standard sed may not support it.

Using shell functions instead of files in the below reproducer so it can be copied-and-pasted to test without needing to create data files first:

getListOfWords() {
  printf '%s\n' I to or the your not so that they and you with Thanks
}

getInFile() {
  cat <<EOF
I would love to try or hear the sample audio your app can produce. I do not want to purchase, because I've purchased so many apps that say they do something and do not deliver.  

Can you please add audio samples with text you've converted? I'd love to see the end results.

Thanks!
EOF
}

sed -E 's/\<('"$(tr '\n' '|' < <(getListOfWords) )"')\>//gI' <(getInFile)

...emits as output:

 would love  try  hear  sample audio  app can produce.  do  want  purchase, because 've purchased  many apps  say  do something  do  deliver.  

Can  please add audio samples  text 've converted? 'd love  see  end results.

!

...which matches your intended output with the sole exception that that stated intended output has some extra behavior around removal of punctuation that the original code makes no attempt to implement.

There is a way to pipe sed commands to sed using sed -f-:

sed 's|^|s/|; s|$|\\s*//gI|' listOfWords.txt | sed -f- file.txt > output.txt

This turns listOfWords.txt into sed replacement commands and pipes them to sed:

  • replaces the start of each line in listOfWords with s/
  • replaces the end of each line in listOfWords with \s*//gI
  • This results in s/word\s*//gI foreach word in listOfWords
  • pipe the list of replacements to sed -f- file.txt, where - means "stdin"

Somewhat similar in concept to OP's attempt with tr, but using only sed as enabled by sed -f-

Here is a Perl alternative. The first argument needs to be the listOfWords file.

perl -pe 'BEGIN {open F, shift; $w=join("|", <F>); $w=~s/\n//g;}
          s/\b($w)\b\s*//g;'  /tmp/listOfWords.txt  /tmp/file.txt

The \s* at the end of the regex let's it also remove trailing spaces to avoid multiple consecutive spaces.

The output with your example is:

would love try hear sample audio app can produce. do want purchase, because 've purchased many apps say do something do deliver.  

Can please add audio samples text 've converted? 'd love see end results.

!

If you also want to remove the ' after "I" and "you", you could add I' and you' at the start of your word list file.

If GNU ed is available/acceptable.

#!/usr/bin/env bash

ed -s input.txt < <(
  printf '%s\n' ',s/^/\\b/' ',s/$/\\b/' '1,$-1s/$/\\|/' '1;$j' 's/^/,s\//' 's/$/\/\/g/' '$a' ,p w . ,p Q  |
  ed -s listOfWords.txt
)
Related