Extract multiple independent regex matches per line

user_8971922 0
Viewed 1360

For the file below, I want to extract the two strings following "XC:Z:" and "XM:Z:". For example:

  • 1st line output should be this: "TGGTCGGCGCGT, GAGTCCGT"
  • 2nd line output should be this: "GAAGCCGCTTCC, ACCGACGG"

The original version of the file has a few more columns and millions of rows than the following example, but it should give you the idea:

    MOUSE_10        XC:Z:TGGTCGGCGCGT       RG:Z:A  XM:Z:GAGTCCGT   ZP:i:33
    MOUSE_10        XC:Z:GAAGCCGCTTCC       NM:i:0  XM:Z:ACCGACGG   AS:i:16
    MOUSE_10        ZP:i:36 XC:Z:TCCCCGGGTACA       NM:i:0  XM:Z:GGGACGGG   ZP:i:28
    MOUSE_10        XC:Z:CAAATTTGGAAA       RG:Z:A  NM:i:1  XM:Z:GCAGATAG

In addition, each of following criteria would be a bonus but is not mandatory if you can get it to work:

  • use standard bash tools: awk, sed, grep, etc. (no GAWK, csvtools,...)
  • assume we don't know the order in which XC and XM appear (although I'm fairly certain XC is almost first, but I am unsure how to check). In the output, however, the XC-string should always be before the XM-string, if at all possible.

The answers from here awk extract multiple groups from each line come awfully close to it, but whenever I try using match(...) I get a "syntax error near unexpected token" message.

Looking forward to your solutions!

Thanks,

Felix

sed awk bash regex
5 Answers

With sed you can capture non-space characters after XC:Z: and XM:Z:

sed -n 's/.*XC:Z:\([^[:blank:]]*\).*XM:Z:\([^[:blank:]]*\).*/\1, \2/p;' file

You can add a second s command for reversed values:

sed -n 's/.*XC:Z:\([^[:blank:]]*\).*XM:Z:\([^[:blank:]]*\).*/\1, \2/;s/.*XM:Z:\([^[:blank:]]*\).*XC:Z:\([^[:blank:]]*\).*/\1, \2/;p;' file
user_13449610

Following awk solution may help you in same.

awk '
/XC:Z:/{
  match($0,/XC:[^ ]*/);
  num=split(substr($0,RSTART,RLENGTH),a,":");
  match($0,/XM:[^ ]*/);
  num1=split(substr($0,RSTART,RLENGTH),b,":");
  print a[num],b[num1]
}'   Input_file

Output will be as follows.

TGGTCGGCGCGT GAGTCCGT
GAAGCCGCTTCC ACCGACGG
TCCCCGGGTACA GGGACGGG
CAAATTTGGAAA GCAGATAG
user_58665800

another awk

$ awk '{c=p="";                               # need to reset c and p before each line
        for(i=1;i<=NF;i++)                    # for all fields in the line
          if($i~/^XC:Z:/) c=substr($i,6)      # check pattern from the start of field
          else if($i~/^XM:Z:/) p=substr($i,6) # if didn't match check other other pattern 
        if(c && p) print c,p}' file           # if both matched print

TGGTCGGCGCGT GAGTCCGT
GAAGCCGCTTCC ACCGACGG
TCCCCGGGTACA GGGACGGG
CAAATTTGGAAA GCAGATAG

this will print the last matches if there are multiple instances on the same line. Here is another one with slightly different characteristic.

$ awk 'function s(x) {return ($i~x)?substr($i,6):""}
      {c=p="";
       for(i=1;i<=NF;i++) {
         c=c?c:s("^XC:Z:"); p=p?p:s("^XM:Z:");
         if(c && p) 
           {print c,p; next}}}' file

TGGTCGGCGCGT GAGTCCGT
GAAGCCGCTTCC ACCGACGG
TCCCCGGGTACA GGGACGGG
CAAATTTGGAAA GCAGATAG

this will print the last of the repeated match before the first match of the other. It they appear in pairs, will print the first pair.

user_14358690

If we don't know the order in which XC and XM appear You can try this sed

sed -E 'h;s/(XC:Z:.*XM:Z:)//;tA;x;s/(.*XM:Z:)([^[:blank:]]*)(.*XC:Z:)([^[:blank:]]*)(.*)/\4,\2/;b;:A;x;s/(.*XC:Z:)([^[:blank:]]*)(.*XM:Z:)([^[:blank:]]*)(.*)/\2,\4/' infile

explanation :

sed -E '
h
# keep the line in the hold space
s/(XC:Z:.*XM:Z:)//;x;tA
# if XCZ come before XMZ, go to A but before everything restore the pattern space with x
s/(.*XM:Z:)([^[:blank:]]*)(.*XC:Z:)([^[:blank:]]*)(.*)/\4,\2/
# XMZ come before XCZ, get the interresting parts and reorder it
b
# It is all for this line
:A
s/(.*XC:Z:)([^[:blank:]]*)(.*XM:Z:)([^[:blank:]]*)(.*)/\2,\4/
# XCZ come before XMZ, get the interresting parts
' infile
user_85929750

Using POSIX awk, you can only use the string-function match(s,ere) as defined by IEEE Std 1003.1-2008 :

match(s, ere)

Return the position, in characters, numbering from 1, in string s where the extended regular expression ere occurs, or zero if it does not occur at all. RSTART shall be set to the starting position (which is the same as the returned value), zero if no match is found; RLENGTH shall be set to the length of the matched string, -1 if no match is found.

The patterns you want to match are XM:Z:[^[:blank:]]* and XC:Z:[^[:blank:]]*. This however assumes you do not have any string which contains something like PXM:Z: (i.e. an extra non-blank character advancing the searched string). When the pattern is found in the line $0, then you only need to extract the important parts, which start 5 characters later.

The following code does the above:

   awk '{match($0,/XM:Z:[^[:blank:]]*/);xm=substr($0,RSTART+5,RLENGTH-5)}
        {match($0,/XC:Z:[^[:blank:]]*/);xc=substr($0,RSTART+5,RLENGTH-5)}
        {print xc","xm}' <file>

As you can see, the first line extracts XM, the second XC and the third prints the outcome with comma-separator ",".

Remark - The following assumptions are made here :

  • each line contains both an xm and xc string
  • no strings of the type [^[:blank:]]X[CM]:Z:[^[:blank:]]* exist

If you are willing to use gawk, then you could use the patsplit function for string operations (Ref. here). You can do this with a single regex /X[CM]:Z:[^[:blank:]]*/. This gives you directly the requested strings in a single call which include the XM:Z: or XM:C: part. Afterwards you can easily sort them and extract the last parts.

The following lines do exactly the same in gawk

   gawk '{patsplit($0,a,/X[MC]:Z:[^[:blank:]]*/) }
         {xc=(a[1]~/^XC/)?a[1]:a[2]; xm=(a[1]~/^XC/)?a[2]:a[1]}
         {print substr(xc,5)","substr(xm,5)' <file>

Nonetheless, I believe the awk solution is cleaner from a symmetric point of view.

user_83440600
Related