Using grep or sed to substitute words not matching a pattern

Viewed 1041

I am new to the regular expression paradigm and I've encountered a problem I'm trying to solve, with no success.

Imagine a file test.txt with:

hello everyone, whatsi up
i hope my program worksa
if it doesnt... ho well!

I would like to output to another file output.txt only the words that start with a consonant and end with a vowel so it would result in:

hello whatsi
hope worksa
ho

I am using grep -o '\b[^ aeiouAEIOU]\w*[aeiouAEIOU]\b' test.txt > output.txt however, the -o flag outputs every matched string onto a new line. What should I do to get the format that I want? Another valid option would be to substitute everything that doesn't match that pattern with a blank space using sed but I failed to do it as well. Should I be using sed or awk instead?

Thanks

7 Answers

With GNU awk for multi-char RS, RT, and word boundaries:

$ gawk -v RS='\\<[^aeiou][[:alpha:]]*[aeiou]\\>' 'RT{print RT}' file
hello
whatsi
hope
worksa
ho

If you need to preserve the original lines breaks then using GNU awk for FPAT instead of RS would be:

$ gawk -v FPAT='\\<[^aeiou][[:alpha:]]*[aeiou]\\>' '{for (i=1; i<=NF; i++) printf "%s%s", $i, (i<NF?OFS:ORS)}' file
hello whatsi
hope worksa
ho

You can instruct grep to treat the input as a set of null-byte-terminated lines, i.e. as one long line if your input does not contain null bytes, with the -z/--null-data flag.

That way you can preserve newline characters (note ...| |\n in the end):

$ grep -Pozi '\b[bcdfghjklmnpqrstvwxyz]\w*[aeiou]\b| |\n' file
hello  whatsi 
 hope   worksa
   ho

but at the expense of null-byte (\x0) characters present (and multiple spaces, due to our regex). Those can be fixed with a few sed expressions:

$ grep -Pozi '\b[bcdfghjklmnpqrstvwxyz]\w*[aeiou]\b| |\n' file \
  | sed -E -e 's/\x0//g' -e 's/ +/ /g' -e 's/^ //' -e 's/ $//'
hello whatsi
hope worksa
ho

(one to strip null-bytes, one to replace multiple spaces with a single space, and two to strip leading and trailing spaces).

Use the -n option to output line number, you can then regroup the matches.

For example, in Perl:

grep -no '\b[^ aeiouAEIOU]\w*[aeiouAEIOU]\b' test.txt \
| perl -aF: -nwE 'chomp $F[1];
                push @{ $b[ $F[0] ] }, $F[1]
                }{ say "@$_" for grep defined, @b'

Perl by itself will work well here: for each line, find each word matching the criteria

perl -lane 'print join " ", grep {/\b[bcdfghjklmnpqrstvwxyz][[:alpha:]]*[aeiou]\b/i} @F' file
  • using -a splits the line into words, stored in the array @F
  • grep will filter only the words matching the regular expression
  • then join the resulting list with spaces and print them out.
    • if no words on a line match, then an empty line will be printed.

Golfing a touch

perl -lape'$_="@{[grep{/\b(?=[a-z])[^aeiou][a-z]*[aeiou]\b/i}@F]}"' file

Note that a digit matches [^aeiou] which is why I added the lookahead (?=[a-z]) to restrict the first character of the word to be alphabetic but not a vowel.

with the help of -P in grep


~ ❱ grep -Po '\w+' file
hello
everyone
whatsi
up
i
hope
my
program
worksa
if
it
doesnt
ho
well
~ ❱ grep -Po '\b(?![oauie])[a-z]+((?=[oauie]).)\b' file
hello
whatsi
hope
worksa
ho
~ ❱ 
~ ❱ # return in a single line:
s~ ❱ grep -zPo '\b(?![oauie])[a-z]+((?=[aeiou]).) \b' file
hello whatsi hope ho ~❱
~ ❱ 
~ ❱

how it works

-P is for using PCRE

and the pattern follows these steps:

  1. it does not match any [aieuo] at the beginning of a word
  2. then matches some characters [a-z]+ if:
  3. the end of that word has [aieuo]

NOTE

My answer does NOT keep the words in its line. I wanted to write a Perl one-liner and then noticed @glenn jackman already did this. Thus you can use that answer or:

~ ❱ perl -lae' print for "@{[ grep{/\b(?![oauie])[a-z]+((?=[oauie]).)\b/} @F ]}" ' file                                                                                                       
hello whatsi                                                                                                                                                                                           
hope worksa
ho
~ ❱ 

or without "@{[ ... ]} operators:

~ ❱ perl -lae '@arr = grep /\b(?![oauie])[a-z]+((?=[oauie]).)\b/, @F;print "@arr"' file
hello whatsi
hope worksa
ho
~ ❱

Following awk solution could help you in same too.

awk '{for(i=1;i<=NF;i++){if(tolower($i) ~ /^[^aeiou].*[aeiou]$/){val=val?val OFS $i:$i}};print val;val=""}'  Input_file

Output will be as follows.

hello whatsi
hope worksa
ho

Adding a non-one liner form with explanation too here.

awk '{
for(i=1;i<=NF;i++){       ##Starting a for loop here which starts from variable i value from 1 to till the value of NF(number of fields) value.
  if(tolower($i) ~ /^[^aeiou].*[aeiou]$/){ ##checking here condition if a field value in lower is satisfying the regex where I am checking if a value NOT starts from vowel and it is ending with vowels.
    val=val?val OFS $i:$i ##Creating a variable named val which will have value of current field value and it will concatenate its own value.
}
};
  print val;              ##Outside of loop, I am printing the value of variable val here, which will have all those words which are satisfying your conditions.
  val=""                  ##Nullifying the value of variable val here.
}
' Input_file              ##Mentioning the Input_file name here.

Here is the regex to match words that start with a consonant and end with a vowel

/\<[^ aeiouAEIOU]\w*[aeiouAEIOU]\>/

which we can use to select our words and remove everything else using Ex/Vim editor.

So given the test.txt file created by the following command:

$ printf "hello everyone, whatsi up\ni hope my program worksa\nif it doesnt... ho well!" > test.txt

this shell command will read the file and save parsed output into out.txt file:

$ ex -s +'%s/\<\w\+\>\(\<[^ aeiouAEIOU]\w*[aeiouAEIOU]\>\)\@<!\s\?//g' +"%s/\([[:punct:]]\+\)//g" +%p +'wq! out.txt' test.txt 
hello  whatsi 
hope worksa
 ho 

Explanation:

  • \<\w\+\> - selects all words;
  • \(\<[^ aeiouAEIOU]\w*[aeiouAEIOU]\>\) - selects our words to keep;
  • \@<! - it matches with zero width if the preceding atom does NOT match just before what follows (see: :help \@<!);
  • %s/pattern/replace/g - substitutes the pattern with replace text;
  • %s/\([[:punct:]]\+\)//g - removes all punctuation characters;
  • +%p - prints the file buffer to the standard output;
  • wq! file.txt - writes the current buffer to the file;

Above solution is based on this answer: How to remove all words which doesn't match the pattern?

Related