Deleting the un-matched portion using sed

user_8270581 0
Viewed 81

I'm having a text file containing data in the following format:

2020-01-01 00:00:00 @gibberish - key1:{value1}, unwanted key2:{value2}, unwanted key3:{value3}

I wanted to collect the timestamp in the beginning and key-value pairs alone. Like the following

2020-01-01 00:00:00,key1:{value1},key2:{value2},key3:{value3}

I'm able to write a regex script that can select the required values (works in visual studio code)

^([0-9 :-]+)|([0-9A-z,_-]+):\{(.*?)\}

(first pattern selects the timestamp and second part selects the key-value pattern)

Now, how can I select the un-matched part and delete it using sed ?

Note: I tried using egrep to match the required pattern and writing it to a new file. But every matched string is written on a new line instead of maintaining on the same line. That is not useful to me.

egrep -o '^([0-9 :-]+)|([0-9A-z,_-]+):\{(.*?)\}' source.txt > target.txt
sed grep macos linux regex
4 Answers

Going from last to first, I can comment that:

  • egrep: yes, that is the designed behavior - egrep is probably not what you want to use.
  • sed: it is important to note that sed uses POSIX regular expressions which is simpler and much more limited than what people expect from regular expressions these days. Most of the new style (enhanced, perl-compatible, etc) regular expression work in the last few decades was done in Perl, which is readily available on UNIX systems and is probably what you want to use (but also note that in macOS, like all Apple distributed UNIX programs, the perl binary there is pretty outdated. It will probably still do what you want, but be warned).
  • Your regular expression uses a range [A-z], which is weird and doesn't work in my egrep or sed - I understand what you want to do, but it shouldn't work in system that actually use character sets (I'm not sure what Visual Studio is doing with this range, but it seems bonkers to me). You probably meant to use [A-Za-z].

I would have written this thing, using Perl, like so:

perl -nle '@res = (); while(m/^([0-9 :-]+\d)|([0-9A-Za-z,_-]+:\{[^}]+\})/g) {
    push @res, "$1$2";
  };
  print join ",",@res' < source.txt > target.txt
user_535380

With your shown samples, could you please try following. Written and tested in GNU awk in case you are ok with it.

awk '
match($0,/[0-9]{4}-[0-9]{2}-[0-9]{2}[[:space:]]+([0-9]{2}:){2}[0-9]{2}/){
  val=""
  printf("%s ",substr($0,RSTART,RLENGTH))
  while(match($0,/key[0-9]+:{value[0-9]+}(,|$)/)){
    val=(val?val OFS:"")substr($0,RSTART,RLENGTH)
    $0=substr($0,RSTART+RLENGTH)
  }
  print val
}
'  Input_file
user_58665800

This might work for you (GNU sed):

sed -E 's/\S+/\n&/3g;s#.*#echo "&"|sed "1b;/:{.*}/!d;s/, *$//"#e;s/ *\n/,/g' file

Split each line into a lines of tokens (keeping the date and time as the first of these lines).

Remove any line (apart from the first) that does not contain the pattern :{...}.

Flatten the lines by replacing the introduced newlines by , separator.

user_9674920 
sed -rn 's/([[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}[[:space:]]([[:digit:]]{2}:){2}[[:digit:]]{2})(.*)(key1.*,)(.*)(key2.*,)(.*)(key3.*$)/\1,\4\6\8/p' <<< "2020-01-01 00:00:00 @gibberish - key1:{value1}, unwanted key2:{value2}, unwanted key3:{value3}"

Enable regular expression interpretation with sed -r or -E and then split the string into 8 sections using parenthesis. Substitute the line for the 1st, 4th, 6th and 8th sections and print.

user_78404400
Related