GNU awk, FPAT and matching negative string regex with regex and special chars

user_4162356 0
Viewed 92

TL (see TL;DR near the end of the question)

I came about this data with pipes as field delimiters (|) and backslash-quote pairs as quotes (\") to fields with delimiters in the data, such as:

1|\"2\"|\"3.1|3.2\"|4  # basically 1, 2, 3.1|3.2, 4

that is (in awk):

$1==1
$2==\"2\"
$3==\"3.1|3.2\"
$4==4

I decided to try and use GNU awk's FPAT to solve the field issue since writing a negative match regex to \" didn't seem that bad.

I came about this answer to Regular expression to match a line that doesn't contain a word with a link to (an offsite link) an online generator of negative regular expressions given an input phrase.

As the generator supports only alphanumeric and space characters currently, \" (backslash-quote) was replaced with bq and the generator provided regex:

^([^b]|b+[^bq])*b*$

| was replaced with a p and the data above replaced with:

1pbq2bqpbq3.1p3.2bqp4
1|\"2\"|\"3.1|3.2\"|4  # original for comparision

Sample FPAT from GNU awk documentation (FPAT="([^,]*)|(\"[^\"]+\")") was used to generate an FPAT:

FPAT="([^p]*)|(bq([^b]|b+[^bq])*b*bq)"

and a trial was done:

$ gawk 'BEGIN {
    FPAT="([^p]*)|(bq([^b]|b+[^bq])*b*bq)"
    OFS=ORS
}
{
    print $1,$2,$3,$4
}' data

which output:

1
bq2bq
bq3.1p3.2bq
4

which is right. Replacing pqs with |"s in the program produced:

$ gawk 'BEGIN {
    FPAT="([^|]*)|(b\"([^b]|b+[^b\"])*b*b\")"
    OFS=ORS
}
{
    print $1,$2,$3,$4
}' data

outputs:

1
b"2b"
b"3.1|3.2b"
4

which is still right. However, when replacing bs with \s and adding some escaping, resulted in:

(TL;DR how to fix escaping in below script)

$ gawk 'BEGIN {
    FPAT="([^|]*)|(\\\"([^\\]|\\+[^\\\"])*\\*\\\")"
    OFS=ORS
} 
{
    print $1,$2,$3,$4
}' data

and output fails or differs from the previous:

1
\"2\"
\"3.1
3.2\"

so there is probably something wrong with my \\s but after too many try and errs my head is filled with backslashes and all thoughts pretty much escaped (pun intended). And as the community is all about sharing, I thought to share my headache with you guys.

Edit: Apparently it's got something to do with backslashes in quotes, since if instead of defining FPAT="..." I use GNU awk's strongly typed typing FPAT=@/.../ I get the correct output:

$ gawk 'BEGIN {
    FPAT=@/([^|]*)|(\\\"([^\\]|\\+[^\\\"])*\\*\\\")/
    OFS=ORS
} 
{
    print $1,$2,$3,$4
}' data

Output now:

1
\"2\"
\"3.1|3.2\"
4
awk regex
1 Answers

You seem to be trying to use [^\\\"] to mean not the string \" but it doesn't mean that, it means neither the char \ nor the char ". You need to have a single char to negate in that part of the FPAT regexp so the approach is to convert every \" in the input to a single char that can't be present in the input (I use \n below as that's usually RS but you can use any char that can't be in the record), then split the record into fields, and then restore the \"s before using each individual field:

$ cat tst.awk
BEGIN { FPAT="([^|]*)|(\n[^\n]+\n)" }
{
    gsub(/\\"/,"\n")              # Replace each\" with \n in the record
    $0 = $0                       # Re-split the record into fields
    for (i=1; i<=NF; i++) {
        gsub("\n","\\\"",$i)      # Replace each \n with \" in the field
        print "$"i"=" $i
    }
}


$ awk -f tst.awk file
$1=1
$2=\"2\"
$3=\"3.1|3.2\"
$4=4

If there is no specific char that can't be present in your input then it's easy to manipulate your input such that whatever character you like cannot be present during field splitting (I'm using \n again here but this time it'd work even if your input was multi-line records containing \ns, assuming you set RS appropriately to allow reading of multi-line records):

$ cat tst.awk
BEGIN { FPAT="([^|]*)|(\n[^\n]+\n)" }
{
    gsub(/@/,"@A")
    gsub(/\n/,"@B")
    gsub(/\\"/,"\n")
    $0 = $0
    for (i=1; i<=NF; i++) {
        gsub("\n","\\\"",$i)
        gsub("@B","\n",$i)
        gsub("@A","@",$i)
        print "$"i"=" $i
    }
}


$ awk -f tst.awk file
$1=1
$2=\"2\"
$3=\"3.1|3.2\"
$4=4
user_17450010
Related