awk arrange record order by field name

Viewed 110

I'm working on a bash script to process various LDAP queries into pipe delimited files. Some of the results records do not include all attributes, and the data for each record does not retun in the same attribute order for each record. I've scripted to ensure all records have the 4 necessary attributes, and am now trying use awk to reorder the fields of the output records to all match an established order. Below is a sample set of records I'm looking to process with the first record representing the desired order/column heads.

cn: User ID
displayName: Display Name
LastLoginTime: Last Login
ExpirationDate: Exp Date

cn: mf_mdsa
displayName: NONE
ExpirationDate: 00000000000000
LastLoginTime: 20201220212738

displayName: NONE
cn: mf_cs
ExpirationDate: 00000000000000
LastLoginTime: 20201220212704

displayName: NONE
ExpirationDate: 00000000000000
LastLoginTime: 20181009205555
cn: OPERATOR

ExpirationDate: 00000000000000
displayName: HENNIE VAN DEVENTER
LastLoginTime: 20181030201447
cn: hvdevent

cn: A0S
displayName: LARA EVERWINE
ExpirationDate: 20190612001951
LastLoginTime: 20190313182136

cn: SODS822
ExpirationDate: 00000000000000
displayName: JAMES SIMS
LastLoginTime: 20210104000757

I'm calling an awk script as follows and outputting to a file

BEGIN {FS=": "; OFS="|"}
$1 == "cn" {cn = $2}
$1 == "displayName" {displayName = $2}
$1 == "LastLoginTime" {LastLoginTime = $2}
$1 == "ExpirationDate" {
        print cn,displayName,LastLoginTime,$2}

While the desired reordering seems to be happening, fields values from one record are bleeding over into the next. I'm supposing it has something to do with how the field label and values are being added to the variables to feed the print, but I've not been able to figure out how to process each record correctly before moving to the next

User ID|Display Name|Last Login|Exp Date
mf_mdsa|NONE|Last Login|00000000000000
mf_cs|NONE|20201220212738|00000000000000
mf_cs|NONE|20201220212704|00000000000000
OPERATOR|NONE|20181009205555|00000000000000
A0S|LARA EVERWINE|20181030201447|20190612001951
SODS822|LARA EVERWINE|20190313182136|00000000000000
SODS822|JAMES SIMS|20190313182136|00000000000000
3 Answers

Since order of your fields are different in each set, you should write full record at empty line only using condition !NF (assuming you have a newline after every 4 lines as shown in the question):

cat rec.awk

BEGIN {FS=": "; OFS="|"}
$1 == "cn" {cn = $2}
$1 == "displayName" {displayName = $2}
$1 == "LastLoginTime" {LastLoginTime = $2}
$1 == "ExpirationDate" {expirationDate=$2}
!NF {if (cn != "") print cn,displayName,LastLoginTime,expirationDate; cn=""}
END {if (cn != "") print cn,displayName,LastLoginTime,expirationDate}

Then use it as:

awk -f rec.awk file

User ID|Display Name|Last Login|Exp Date
mf_mdsa|NONE|20201220212738|00000000000000
mf_cs|NONE|20201220212704|00000000000000
OPERATOR|NONE|20181009205555|00000000000000
hvdevent|HENNIE VAN DEVENTER|20181030201447|00000000000000
A0S|LARA EVERWINE|20190313182136|20190612001951

You don't need to manipulate your data in advance to ensure every record has all 4 fields, nor do you need to hard-code field names in your script since you already provide them at the top of your data.

Given this script (using any awk in any shell on every Unix box):

$ cat tst.awk
BEGIN {
    FS = RS
    RS = ""
    OFS = "|"
}
{
    delete tag2val
    for (i=1; i<=NF; i++) {
        tag = val = $i
        sub(/:.*/,"",tag)
        sub(/[^:]*: */,"",val)
        tag2val[tag] = val
        if ( NR == 1 ) {
            tags[++numTags] = tag
        }
    }

    for (tagNr=1; tagNr<=numTags; tagNr++) {
        tag = tags[tagNr]
        val = (tag in tag2val ? tag2val[tag] : "NONE")
        printf "%s%s", val, (tagNr<numTags ? OFS : ORS)
    }
}

and this input:

$ cat file
cn: User ID
displayName: Display Name
LastLoginTime: Last Login
ExpirationDate: Exp Date

cn: mf_mdsa
ExpirationDate: 00000000000000
LastLoginTime: 20201220212738

cn: mf_cs
ExpirationDate: 00000000000000
LastLoginTime: 20201220212704

LastLoginTime: 20181009205555
cn: OPERATOR

cn: hvdevent

ExpirationDate: 20190612001951

displayName: JAMES SIMS
LastLoginTime: 20210104000757

we get this output:

$ awk -f tst.awk file
User ID|Display Name|Last Login|Exp Date
mf_mdsa|NONE|20201220212738|00000000000000
mf_cs|NONE|20201220212704|00000000000000
OPERATOR|NONE|20181009205555|NONE
hvdevent|NONE|NONE|NONE
NONE|NONE|NONE|20190612001951
NONE|JAMES SIMS|20210104000757|NONE

Instead of setting FS=": " you can take advantage of awk's paragraph mode by setting RS= FS='\n' to break a record on \n\n and a field being a line. Then split that line on the :

Since awk arrays are unordered, you need to keep an order index. In this case, the order is determined by the order of the first record. That is easily changed to a different order by assigning such to order instead of reading it from the first record.

Here is an example (perhaps not optimized...)

awk -v RS= -v FS='\n' 'FNR==1 {
                        for(i=1;i<=NF;i++){
                            split($i,x,/:[ ]*/)
                            order[i]=x[2]
                            key[x[2]]=x[1]
                        }
                        len=i-1
                        for(i=1; i<=len; i++) 
                            printf "%s%s", order[i], i<len ? "|" : ORS
                        next
                        }
                        split("",field)
                        {for(i=1;i<=NF;i++) {
                            split($i,x,/:[ ]*/)
                            field[x[1]]=x[2]
                        }
                    for(i=1;i<=len;i++) {
                        printf "%s%s", field[key[order[i]]], i<len ? "|" : ORS
                    }
}' file 

With ruby, it is a little easier. Ruby supports paragraph mode (with the -00 switch), ordered hashes, and multi-line regexs. Therefore that awk can be reduced to:

ruby -00 -lne '
    data=$_.scan(/^([^:]+):[ \t]*(.+)$/).to_h
    if $.==1 
        puts data.values.join("|")
        order=data
    else
        puts order.map {|k,v| data[k]}.join("|")
    end
' file

Either of these prints:

User ID|Display Name|Last Login|Exp Date
mf_mdsa|NONE|20201220212738|00000000000000
mf_cs|NONE|20201220212704|00000000000000
OPERATOR|NONE|20181009205555|00000000000000
hvdevent|HENNIE VAN DEVENTER|20181030201447|00000000000000
A0S|LARA EVERWINE|20190313182136|20190612001951
SODS822|JAMES SIMS|20210104000757|00000000000000
Related