awk command of add column to std over repeated row

Viewed 57

i have a dataset as file:

1   aaa 1
1   aaa 3
1   aaa 5
1   ccc 9
1   ccc 2
1   ccc 2
1   ddd 4
1   ddd 6
1   ddd 8
1   ddd 1
2   ddd 5
2   ddd 12
2   ddd 13

i want to standard deviation of grouped data:

for examles at

1   aaa

have 1,3,5 values. std of these is 2,00

at

2   ddd

have 5,12,13 values. std of these is ~ 4,36

i have tried,

awk 'BEGIN{ FS=OFS="\t" }
  NR==FNR{ 
    a[$1,$2]+=$3
    next
  }
  {delta = $3 - avg; avg += delta / NR; mean2 += delta * ($3 - avg); }
  { $(NF+1)=sqrt(mean2 / NR) }
  1
' file file

Desired output:

1   aaa 1   2,00
1   aaa 3   2,00
1   aaa 5   2,00
1   ccc 9   4,04
1   ccc 2   4,04
1   ccc 2   4,04
1   ddd 4   2,99
1   ddd 6   2,99
1   ddd 8   2,99
1   ddd 1   2,99
2   ddd 5   4,36
2   ddd 12  4,36
2   ddd 13  4,36
1 Answers

The problems in your script are:

  • You are confusing the fixed values (such as average) with the accumulating values (such as sum of delta**2) and calculating them changing the values line by line in the 2nd loop.
  • You need to use each count of the dataset as the denominator, not NR. (To be specific, count-1 to calculate the standard deviation of sample)

Implementing the formula of calculating standard deviation of sample, would you please try:

awk '
  BEGIN { FS=OFS="\t" }
  NR==FNR { 
    a[$1,$2] += $3
    n[$1,$2]++
    next
  }
  # the following block is executed just before processing 2nd "file"
  FNR==1 {
    for (i in a) {
      avg[i] = a[i] / n[i]
    }
  }
  # the following block is executed while reading the 2nd "file"
  { var[$1,$2] += ($3 - avg[$1,$2]) * ($3 - avg[$1, $2]) }
  END {
    for (i in a) {
      str = i; sub("\x1c", "\t", str)   # convert 2-D element separator to a tab
      print str, sqrt(var[i] / n[i] - 1)
    }
  }
' file file

Single loop version (more common):

awk 'BEGIN { FS=OFS="\t" }
  { 
    a[$1,$2] += $3              # sum of xi
    a2[$1,$2] += $3 * $3        # sum of xi * xi
    n[$1,$2]++
  }
  END {
    for (i in a) {
      avg[i] = a[i] / n[i]
      var[i] = (a2[i] - n[i] * avg[i] * avg[i]) / (n[i] - 1)
      if (var[i] < 0) var[i] = 0        # avoid rounding error
      str = i; sub("\x1c", "\t", str)   # convert 2-D element separator to a tab
      print str, sqrt(var[i])
    }
  }
' file

Output:

1   ddd 2.98608
1   aaa 2
2   ddd 4.3589
1   ccc 4.04145

[EDIT]
If you want to append the stdev.s values to the original data set side by side, please try the following:

awk 'BEGIN { FS=OFS="\t" }
  NR==FNR { 
    a[$1,$2] += $3              # sum of xi
    a2[$1,$2] += $3 * $3        # sum of xi * xi
    n[$1,$2]++
    next
  }
  # the following block is executed just before processing 2nd "file"
  FNR==1 {
    for (i in a) {
      avg[i] = a[i] / n[i]
      var[i] = (a2[i] - n[i] * avg[i] * avg[i]) / (n[i] - 1)
      if (var[i] < 0) var[i] = 0        # avoid rounding error
      str = i; sub("\x1c", "\t", str)   # convert 2-D element separator to a tab
      std[i] = sprintf("%.2f", sqrt(var[i]))
    }
  }
  # the following block is executed while reading the 2nd "file"
  { $(NF+1) = std[$1,$2] }
  1
' file file

Output:

1   aaa 1   2.00
1   aaa 3   2.00
1   aaa 5   2.00
1   ccc 9   4.04
1   ccc 2   4.04
1   ccc 2   4.04
1   ddd 4   2.99
1   ddd 6   2.99
1   ddd 8   2.99
1   ddd 1   2.99
2   ddd 5   4.36
2   ddd 12  4.36
2   ddd 13  4.36
Related