Unix: File have 28 columns separated by space, print column 8 unique values with sum of column 16 to column28

Viewed 83

Sample File.txt: enter image description here

"Object_Type" "710" "30" "14" "Partner" "Rpt" "Cool" "0ICPRP900" "Des_Space" "HBBREX" "soon_Type" "FY28" "Jan" "50" "USD_US" #Mi #Mi #Mi 0.006641 0 0 0 0 0 0 0 0 0
"Object_Type" "710" "30" "14" "Partner" "Rpt" "Cool" "0ICPRP 900" "Des_Space" "HBBREX" "soon_Type" "FY28" "Jan" "50" "USD_US" #Mi #Mi #Mi 0.01 0 0 0 0 0 0 0 0 0
"Object_Type" "710" "30" "14" "Partner" "Rpt" "Cool" "0ICPRP A&B_900" "Des_Space" "HBBREX" "soon_Type" "FY28" "Jan" "50" "USD_US" #Mi #Mi #Mi 0.641 0 0 0 0 0 0 0 0 0

Looking for output line:

0ICPRP900 0.006641
0ICPRP 900 0.01
0ICPRP A&B_900 0.641

I have tried below command and it's printed column 8 (column not printed after have space special char )

awk '{a[$8]+=$17+$18+$19+$20+$21+$22+$23+$24+$25+$26+$27+$28}END{for (i in a) 
print i,bc a[i]}' File.txt

Output:

"0ICPRP900" 0.006641
"0ICPRP 0.01
"0ICPRP 0.641

Any suggestions how to get unique column8 (column8 with in "") with sum of column 16 to Column 28

3 Answers

With GNU awk, you could try following. Written and tested with your shown samples.

awk -v FPAT='"[^"]+"|[^"]*' '
{
  sum=0
  num=split($NF,arr," ")
  for(i=1;i<=num;i++){
    sum+=arr[i]
  }
  gsub(/"/,"",$15)
  print $15,sum
}
' Input_file

Explanation: Adding detailed explanation for above.

awk -v FPAT='"[^"]+"|[^"]*' '  ##Starting awk program from here, setting FPAT to set fields as per requirement.
{
  sum=0                        ##Nullifying sum here.
  num=split($NF,arr," ")       ##Splitting last field into array here.
  for(i=1;i<=num;i++){         ##Traversing through all elements of arr here.
    sum+=arr[i]                ##Creating sum from array elements.
  }
  gsub(/"/,"",$15)             ##Removing all " here
  print $15,sum                ##Printing 15 field along with sum here.
}
' Input_file                   ##mentioning Input_file name here.

Output will be as follows:

0ICPRP900 0.006641
0ICPRP 900 0.01
0ICPRP A&B_900 0.641

You really have there is a form of CSV and it would serve well to use a proper parser for CSV. It is a surprisingly complex format...

The lightest weight CSV parser commonly available at the command line is Ruby (but Python, Perl work well too...)

Here is a Ruby:

ruby  -r csv -lane '
   BEGIN{ options={:col_sep=>" ", :quote_char=>"\""} }
   CSV.parse($_, **options)
    { |r| puts "#{r[7]} #{r[18..27].inject(0){|s,x| s+x.to_f}}" }' file

Prints:

0ICPRP900 0.006641
0ICPRP 900 0.01
0ICPRP A&B_900 0.641

Using any awk:

$ awk -F'"' '{n=split($31,f," "); sum=0; for (i=4; i<=n; i++) sum+=f[i]; print $16, sum}' file
0ICPRP900 0.006641
0ICPRP 900 0.01
0ICPRP A&B_900 0.641

Or using GNU awk for FPAT and gensub():

$ awk -v FPAT='"[^"]*"|[^ ]+' '{sum=0; for (i=19; i<=NF; i++) sum+=$i; print gensub(/"/,"","g",$8), sum}' file
0ICPRP900 0.006641
0ICPRP 900 0.01
0ICPRP A&B_900 0.641

or if by unique column8 you mean you want to sum the values for multiple occurrences of the same $8 value across the input file you can do either of:

any awk:

awk -F'"' '
    {
        n = split($31,f," ")
        for (i=4; i<=n; i++) {
            sum[$16] += f[i]
        }
    }
    END {
        for (key in sum) {
            print key, sum[key]
        }
    }
' file

GNU awk:

awk -v FPAT='"[^"]*"|[^ ]+' '
    {
        for (i=19; i<=NF; i++) {
            sum[gensub(/"/,"","g",$8)] += $i
        }
    }
    END {
        for (key in sum) {
            print key, sum[key]
        }
    }
' file
Related