searching for a match then replacing a value after this match with a value from the previous line

Admin 1
Viewed 137

I have multiple files that contain thousands of lines of the following pattern:

O   HOH     1      11.700 -11.906   1.533
H   HOH     0      12.561 -12.318   1.596
H   HOH     0      11.445 -12.031   0.627
O   HOH     2      17.897   8.999  -0.104
H   HOH     0      18.444   8.521   0.516
H   HOH     0      17.127   8.457  -0.223
O   HOH     3      -0.688  10.310 -15.189
H   HOH     0      -0.760   9.366 -15.055
H   HOH     0      -0.494  10.665 -14.324
O   HOH     4     -10.418   3.733   4.069
H   HOH     0      -9.928   3.726   4.897
H   HOH     0     -11.222   4.218   4.268
O   HOH     5       2.655  -8.910  13.338
H   HOH     0       2.377  -9.739  13.737
H   HOH     0       1.847  -8.531  12.992
O   HOH     6      -0.191  -1.010 -19.469
H   HOH     0       0.455  -1.460 -18.924
H   HOH     0      -0.571  -0.343 -18.895
O   HOH     7      14.338 -15.985  -1.467
H   HOH     0      15.129 -15.647  -1.886
H   HOH     0      13.854 -16.408  -2.175
O   HOH     8     -15.510  -7.073   1.857
H   HOH     0     -15.526  -6.116   1.874
H   HOH     0     -16.341  -7.321   1.464
O   HOH     9       9.155   6.561  -7.729
H   HOH     0       9.419   7.335  -7.245
H   HOH     0       9.943   6.018  -7.768
.
.
.
.
O   HOH  999999     9.155   6.561  -7.729
H   HOH     0       9.419   7.335  -7.245
H   HOH     0       9.943   6.018  -7.768

what I need is to replace each 2 sequential instants of 'H HOH 0' with 'H HOH xxx' where xxx is the value right above the 0s

what I have tried was that, I split the big file into 3-line files using 

split -l 3 foo

then using awk I copied the needed value to another file 

awk 'NR==1 print $3' > foo--

then tried to use sed to replace the pattern by reading the file foo-- and replacing the ' 0 ' in foo 

sed -e '/  0  /{r foo--' -e 'd}' foo

but that didn't work

pretty sure there is a better way to do this

The end result should look like this:

O   HOH     1      11.700 -11.906   1.533
H   HOH     1      12.561 -12.318   1.596
H   HOH     1      11.445 -12.031   0.627
O   HOH     2      17.897   8.999  -0.104
H   HOH     2      18.444   8.521   0.516
H   HOH     2      17.127   8.457  -0.223
O   HOH     3      -0.688  10.310 -15.189
H   HOH     3      -0.760   9.366 -15.055
H   HOH     3      -0.494  10.665 -14.324
O   HOH     4     -10.418   3.733   4.069
H   HOH     4      -9.928   3.726   4.897
H   HOH     4     -11.222   4.218   4.268
O   HOH     5       2.655  -8.910  13.338
H   HOH     5       2.377  -9.739  13.737
H   HOH     5       1.847  -8.531  12.992
O   HOH     6      -0.191  -1.010 -19.469
H   HOH     6       0.455  -1.460 -18.924
H   HOH     6      -0.571  -0.343 -18.895
.
.
.
O   HOH     99999      -0.191  -1.010 -19.469
H   HOH     99999       0.455  -1.460 -18.924
H   HOH     99999      -0.571  -0.343 -18.895
sed awk bash
5 Answers

You can do what you need in awk, but keeping to format nice and tidy will require using FIELDWIDTHS and fprinf. The problem is due to awk stripping leading and trailing whitespace when it must recompute the record, such as when the record is changed. Unchanged records are not modified in this way. This creates a challenge when providing tabular output.

To use the FIELDWIDTHS of each field, provide a list of the widths in the BEGIN rule. Example:

awk '
    BEGIN  { FIELDWIDTHS = "1 6 8 10 8 8" }
    $1 == "O" {n = $3}
    $1 == "H" {$3 = n}
    {printf "%s%6s%8s%10s%8s%8s\n", $1,$2,$3,$4,$5,$6}
' file

(note: FIELDWIDTHS is a GNU awk feature)

With your data, removing the ..., you can just copy/middle-mouse-paste at the command line to test, e.g.

$ awk '
>     BEGIN  { FIELDWIDTHS = "1 6 8 10 8 8" }
>     $1 == "O" {n = $3}
>     $1 == "H" {$3 = n}
>     {printf "%s%6s%8s%10s%8s%8s\n", $1,$2,$3,$4,$5,$6}
> ' file
O   HOH     1      11.700 -11.906   1.533
H   HOH     1      12.561 -12.318   1.596
H   HOH     1      11.445 -12.031   0.627
O   HOH     2      17.897   8.999  -0.104
H   HOH     2      18.444   8.521   0.516
H   HOH     2      17.127   8.457  -0.223
O   HOH     3      -0.688  10.310 -15.189
H   HOH     3      -0.760   9.366 -15.055
H   HOH     3      -0.494  10.665 -14.324
O   HOH     4     -10.418   3.733   4.069
H   HOH     4      -9.928   3.726   4.897
H   HOH     4     -11.222   4.218   4.268
O   HOH     5       2.655  -8.910  13.338
H   HOH     5       2.377  -9.739  13.737
H   HOH     5       1.847  -8.531  12.992
O   HOH     6      -0.191  -1.010 -19.469
H   HOH     6       0.455  -1.460 -18.924
H   HOH     6      -0.571  -0.343 -18.895
O   HOH     7      14.338 -15.985  -1.467
H   HOH     7      15.129 -15.647  -1.886
H   HOH     7      13.854 -16.408  -2.175
O   HOH     8     -15.510  -7.073   1.857
H   HOH     8     -15.526  -6.116   1.874
H   HOH     8     -16.341  -7.321   1.464
O   HOH     9       9.155   6.561  -7.729
H   HOH     9       9.419   7.335  -7.245
H   HOH     9       9.943   6.018  -7.768
O   HOH  999999     9.155   6.561  -7.729
H   HOH  999999     9.419   7.335  -7.245
H   HOH  999999     9.943   6.018  -7.768
user_34221020

With GNU awk:

awk '$1=="O"{p=$3}; $1=="H" {$3=p} {printf("%s %5s      %s %10.3f %7.3f %7.3f\n",$1,$2,$3,$4,$5,$6)} ' file

Output:

O   HOH      1     11.700 -11.906   1.533
H   HOH      1     12.561 -12.318   1.596
H   HOH      1     11.445 -12.031   0.627
O   HOH      2     17.897   8.999  -0.104
H   HOH      2     18.444   8.521   0.516
H   HOH      2     17.127   8.457  -0.223
O   HOH      3     -0.688  10.310 -15.189
H   HOH      3     -0.760   9.366 -15.055
H   HOH      3     -0.494  10.665 -14.324
O   HOH      4    -10.418   3.733   4.069
H   HOH      4     -9.928   3.726   4.897
H   HOH      4    -11.222   4.218   4.268
O   HOH      5      2.655  -8.910  13.338
H   HOH      5      2.377  -9.739  13.737
H   HOH      5      1.847  -8.531  12.992
O   HOH      6     -0.191  -1.010 -19.469
H   HOH      6      0.455  -1.460 -18.924
H   HOH      6     -0.571  -0.343 -18.895
O   HOH      7     14.338 -15.985  -1.467
H   HOH      7     15.129 -15.647  -1.886
.
.
.
O   HOH      999999      9.155   6.561  -7.729
H   HOH      999999      9.419   7.335  -7.245
H   HOH      999999      9.943   6.018  -7.768
user_37768580

To just get the values you want if you don't care about spacing between fields:

$ awk '$1=="H"{$3=p} {p=$3} 1' file
O   HOH     1      11.700 -11.906   1.533
H HOH 1 12.561 -12.318 1.596
H HOH 1 11.445 -12.031 0.627
O   HOH     2      17.897   8.999  -0.104
H HOH 2 18.444 8.521 0.516
H HOH 2 17.127 8.457 -0.223
O   HOH     3      -0.688  10.310 -15.189
H HOH 3 -0.760 9.366 -15.055
H HOH 3 -0.494 10.665 -14.324
O   HOH     4     -10.418   3.733   4.069
H HOH 4 -9.928 3.726 4.897
H HOH 4 -11.222 4.218 4.268
O   HOH     5       2.655  -8.910  13.338
H HOH 5 2.377 -9.739 13.737
H HOH 5 1.847 -8.531 12.992
O   HOH     6      -0.191  -1.010 -19.469
H HOH 6 0.455 -1.460 -18.924
H HOH 6 -0.571 -0.343 -18.895
O   HOH     7      14.338 -15.985  -1.467
H HOH 7 15.129 -15.647 -1.886
H HOH 7 13.854 -16.408 -2.175
O   HOH     8     -15.510  -7.073   1.857
H HOH 8 -15.526 -6.116 1.874
H HOH 8 -16.341 -7.321 1.464
O   HOH     9       9.155   6.561  -7.729
H HOH 9 9.419 7.335 -7.245
H HOH 9 9.943 6.018 -7.768
O   HOH  999999     9.155   6.561  -7.729
H HOH 999999 9.419 7.335 -7.245
H HOH 999999 9.943 6.018 -7.768

or space-separated with the last 4 fields right-aligned:

$ awk '$1=="H"{$3=p} {p=$3} 1' file | column -t -R3,4,5,6
O  HOH       1   11.700  -11.906    1.533
H  HOH       1   12.561  -12.318    1.596
H  HOH       1   11.445  -12.031    0.627
O  HOH       2   17.897    8.999   -0.104
H  HOH       2   18.444    8.521    0.516
H  HOH       2   17.127    8.457   -0.223
O  HOH       3   -0.688   10.310  -15.189
H  HOH       3   -0.760    9.366  -15.055
H  HOH       3   -0.494   10.665  -14.324
O  HOH       4  -10.418    3.733    4.069
H  HOH       4   -9.928    3.726    4.897
H  HOH       4  -11.222    4.218    4.268
O  HOH       5    2.655   -8.910   13.338
H  HOH       5    2.377   -9.739   13.737
H  HOH       5    1.847   -8.531   12.992
O  HOH       6   -0.191   -1.010  -19.469
H  HOH       6    0.455   -1.460  -18.924
H  HOH       6   -0.571   -0.343  -18.895
O  HOH       7   14.338  -15.985   -1.467
H  HOH       7   15.129  -15.647   -1.886
H  HOH       7   13.854  -16.408   -2.175
O  HOH       8  -15.510   -7.073    1.857
H  HOH       8  -15.526   -6.116    1.874
H  HOH       8  -16.341   -7.321    1.464
O  HOH       9    9.155    6.561   -7.729
H  HOH       9    9.419    7.335   -7.245
H  HOH       9    9.943    6.018   -7.768
O  HOH  999999    9.155    6.561   -7.729
H  HOH  999999    9.419    7.335   -7.245
H  HOH  999999    9.943    6.018   -7.768

or in TSV format:

$ awk -v OFS='\t' '{$3=($1=="H" ? p : $3); p=$3} 1' file
O       HOH     1       11.700  -11.906 1.533
H       HOH     1       12.561  -12.318 1.596
H       HOH     1       11.445  -12.031 0.627
O       HOH     2       17.897  8.999   -0.104
H       HOH     2       18.444  8.521   0.516
H       HOH     2       17.127  8.457   -0.223
O       HOH     3       -0.688  10.310  -15.189
H       HOH     3       -0.760  9.366   -15.055
H       HOH     3       -0.494  10.665  -14.324
O       HOH     4       -10.418 3.733   4.069
H       HOH     4       -9.928  3.726   4.897
H       HOH     4       -11.222 4.218   4.268
O       HOH     5       2.655   -8.910  13.338
H       HOH     5       2.377   -9.739  13.737
H       HOH     5       1.847   -8.531  12.992
O       HOH     6       -0.191  -1.010  -19.469
H       HOH     6       0.455   -1.460  -18.924
H       HOH     6       -0.571  -0.343  -18.895
O       HOH     7       14.338  -15.985 -1.467
H       HOH     7       15.129  -15.647 -1.886
H       HOH     7       13.854  -16.408 -2.175
O       HOH     8       -15.510 -7.073  1.857
H       HOH     8       -15.526 -6.116  1.874
H       HOH     8       -16.341 -7.321  1.464
O       HOH     9       9.155   6.561   -7.729
H       HOH     9       9.419   7.335   -7.245
H       HOH     9       9.943   6.018   -7.768
O       HOH     999999  9.155   6.561   -7.729
H       HOH     999999  9.419   7.335   -7.245
H       HOH     999999  9.943   6.018   -7.768

If you need to retain the original spacing, though, then with GNU awk for the 4th arg to split() this will retain whatever spacing you started with:

$ cat tst.awk
{
    if ($1 == "H") {
        split(prev,pflds,FS,pseps)
        n = split($0,flds,FS,seps)
        seps[2] = pseps[2]
        flds[3] = pflds[3]
        seps[3] = sprintf("%*s",length(pseps[3]) + length(pflds[4]) - length(flds[4]),"")
        $0 = ""
        for (i=1; i<=n; i++) {
            $0 = $0 flds[i] seps[i]
        }
    }
    print
    prev = $0
}

.

$ awk -f tst.awk file
O   HOH     1      11.700 -11.906   1.533
H   HOH     1      12.561 -12.318   1.596
H   HOH     1      11.445 -12.031   0.627
O   HOH     2      17.897   8.999  -0.104
H   HOH     2      18.444   8.521   0.516
H   HOH     2      17.127   8.457  -0.223
O   HOH     3      -0.688  10.310 -15.189
H   HOH     3      -0.760   9.366 -15.055
H   HOH     3      -0.494  10.665 -14.324
O   HOH     4     -10.418   3.733   4.069
H   HOH     4      -9.928   3.726   4.897
H   HOH     4     -11.222   4.218   4.268
O   HOH     5       2.655  -8.910  13.338
H   HOH     5       2.377  -9.739  13.737
H   HOH     5       1.847  -8.531  12.992
O   HOH     6      -0.191  -1.010 -19.469
H   HOH     6       0.455  -1.460 -18.924
H   HOH     6      -0.571  -0.343 -18.895
O   HOH     7      14.338 -15.985  -1.467
H   HOH     7      15.129 -15.647  -1.886
H   HOH     7      13.854 -16.408  -2.175
O   HOH     8     -15.510  -7.073   1.857
H   HOH     8     -15.526  -6.116   1.874
H   HOH     8     -16.341  -7.321   1.464
O   HOH     9       9.155   6.561  -7.729
H   HOH     9       9.419   7.335  -7.245
H   HOH     9       9.943   6.018  -7.768
O   HOH  999999     9.155   6.561  -7.729
H   HOH  999999     9.419   7.335  -7.245
H   HOH  999999     9.943   6.018  -7.768
user_17450010

This should work:

awk '{ if($1=="H" && $2=="HOH" && $3=="0") $3=val; else val=$3 } 1'

Alternatively, to preserve field widths:

awk '{
    if($1=="H" && $2=="HOH" && $3=="0")
        sub(sprintf("0 {1,%d}", length(val)), val " ")
    else
        val=$3
} 1'
user_28019130

This might work for you (GNU sed):

sed -E '/^H\s+HOH\s+0\s/!{h;b};N;/\nH\s+HOH\s+0\s/{G;s/0(.*\n.[^0]*)0(.*)\n\S\s+\S+\s+(\S+).*/\3\1\3\2/};P;D' file

If a line does not begin H HOH 0, print as normal and retain a copy in the hold space. Otherwise, append the following line and if that too begins H HOH 0, then append the copy from the hold space and using substitution rearrange the first two lines to the desired format. In all other cases, print the next line and repeat.

In more detail, as requested:

  • /^H\s+HOH\s+0\s/!{h;b} If a line does not begin H HOH 0, print as normal and retain a copy in the hold space (h copies to the hold space, b breaks out of the sed command cycle, prints the current line in the pattern space and if available, fetches the next line).
  • N The current line begins H HOH 0 append the next line.
  • /\nH\s+HOH\s+0\s/ The appended line also begins H HOH 0 so:
  • G Append the copy of the hold space i.e. the last line that did not begin H HOH 0, and:
  • s/0(.*\n.[^0]*)0(.*)\n\S\s+\S+\s+(\S+).*/\3\1\3\2/ Substitute the 3rd fields in both the last two lines with the 3rd field from the copy of the hold space.
  • P Print upto and including the first newline.
  • D Delete upto and including the first newline.

The last two sed commands cater for the case where the line appended to the pattern space does not match the first three fields of the second. This allows the first of those lines to be printed as normal and the second to be treated as if that line had been presented afresh.

user_9674920
Related