Extract all ocurrences between delimiters being that they can appear more than one per line

user_12364348 0
Viewed 102

I have a document file.yaml that have something like placeholders to replace:

class: ##TOPIC##-area
  name: myClass
type: Class
secretKey: private-##SECRET_KEY##

so far I've used grep to get the values of placeholders

grep -P '(?<=##).*(?=##)' file.yaml

then, I had those values:

TOPIC
SECRET_KEY

now, we have to introduce new properties that can have more than one placeholder per line

class: ##TOPIC##-area
  name: myClass
type: Class
secretKey: private-##SECRET_KEY##-encoded-##SUFFIX_CODE##

hence, grep no longer worked because the output became:

TOPIC
SECRET_KEY##-encoded-##SUFFIX_CODE

but, I want to have

TOPIC
SECRET_KEY
SUFFIX_CODE

I accept all kinds of suggestions and ideas to solve that. thanks

edit: the idea is to just get those placeholders, not replace them. sorry for the misunderstanding.

grep awk shell yaml bash
7 Answers

1st solution: With using GNU awk, please try following awk code. Simple explanation would be, setting RS(record separator) as ## till next occurrence of # followed by 2 occurrences of ##. Then simply printing matched lines only by removing the not needed ## in outcome as per shown output samples.

awk -v RS='##[^#]*##' 'RT{print substr(RT,3,length(RT)-4)}' Input_file

2nd solution: With any awk, please try following program. Simple explanation would be, using match function of awk to match regex /##[^#]*##/(explained in above first solution already); in a while loop to print all the matches found in each line.

awk '{while(match($0,/##[^#]*##/)){print substr($0,RSTART+2,RLENGTH-4);$0=substr($0,RSTART+RLENGTH)}}' Input_file
user_58665800

If Perl is your option, would you please try:

perl -lne 'print $1 while /##(.+?)##/g' file.yaml
  • -l option appends newline character to the output of print.
  • -ne option is mostly similar to that of sed.
  • $1 refers to the capture group 1 of the regex.
  • The while /pattern/g syntax enables the multiple matches in the same line.
user_85723800

When you want to use grep, try something like

grep -Eo "##[^#]*##" file.yaml | tr -d '#'

With awk you can have a multi-character separator, that looks easier:

awk -F'##' '{for (i=2; i<=NF;i+=2) {print $i}}' file.yaml
user_32201130

Use a negative lookahead to exclude substrings that contain ## in them from the match.

(?<=##)((?!##).)*(?=##)

DEMO

Note that this will also return -encoded- since it's also between a pair of ##. Normally you won't get overlapping matches, but lookarounds aren't considered part of the match, so they don't count as overlapping.

user_14918950

The reason you get that output, is because the .* is greedy. You can make it non greedy .*? instead.

If you only want to output the matches, you can add -o to grep.

grep -oP '(?<=##).*?(?=##)' file.yaml

Output

TOPIC
SECRET_KEY
-encoded-
SUFFIX_CODE

If you have only uppercase placeholders where there can be an underscore in between, you can make the pattern a bit more specific.

grep  -oP "##\K[A-Z]+(?:_[A-Z]+)*(?=##)" file.yaml
  • ## Match literally
  • \K Forget what is matched so far (Clear the current match buffer)
  • [A-Z]+(?:_[A-Z]+)* Match 1+ uppercase chars, optionally repeated by _ and uppercase chars
  • (?=##) Positive lookahead, assert ## to the right

See a regex demo.

Output

TOPIC
SECRET_KEY
SUFFIX_CODE
user_54249880

If sed is an option, you can try this solution:

sed '/##.*##/!d
     s/##/\
/
     s/.*\n//
     s/##/\
/
     P;D' file

or, a plain bash solution:

#!/bin/bash

while read -r line; do
    while [[ $line = *'##'*'##'* ]]; do
        line=${line#*'##'}
        printf '%s\n' "${line%%'##'*}"
        line=${line#*'##'}
    done
done < file
user_138090010

Using GNU awk for multi-char RS, RT:

awk -v RS='##[[:upper:]_]+##' 'RT{gsub(/##/,"",RT);print RT}' file
TOPIC
SECRET_KEY
SUFFIX_CODE
user_142594650
Related