sed: search and replace values at position for last line of file

Viewed 623

I have a fixed width file where the last line of the file needs to be checked at a certain position, and if value is found at that position, it is changed to another value specified by an ARRAY. The input file looks like this for the last line

210204154828000999990000056000000072328100000540000000

The values of interest are at position 27, 40 and 47. The values at those positions are 6, 1, 4 respectively. I'm using an ARRAY for checking these values. If the position of the ARRAY matches the value of the position, replace with the value of the ARRAY location. The code looks like this

#!/bin/bash

chars=( { A B C D E F G H I )

for num in "${!chars[@]}" 
do  
    sed -i -E '$ s%^(.{26})$num%\1${chars[$num]}%' $1
done

I don't get a failure when running this but I don't see it updating the last line either. The expected output if this were working correctly would be the following:

21020415482800099999000005F000000072328A000005D0000000

The command to execute is as follows

# ./convert.sh file1.txt

I'm sure something is wrong with my sed but I just can't figure it out. Any help would be appreciated.

8 Answers

You don't want to have to process the file 10 times for each index.

Here's a single call to awk

tac file | awk '
    BEGIN {
        split("A B C D E F G H I", chars); chars[0] = "{"
        split("27 40 47", idx)
    }
    FNR == 1 {
        line = $0
        for (i in idx) {
            char = substr(line, idx[i], 1)
            repl = chars[char]
            line = substr(line, 1, idx[i]-1) repl substr(line, idx[i]+1)
        }
        $0 = line
    }
    1
' | tac

That reverses the file, operates on the first line, then re-reverses the file


To save inplace, use sponge from the moreutils package:

tac file | awk '...' | tac | sponge file

or use a temp file

tmp=$(mktemp)
tac file | awk '...' | tac > "$tmp" && mv "$tmp" file

The equivalent perl is unsurprisingly more concise

tac file | perl -pe '
    BEGIN { @c = qw/ { A B C D E F G H I /; }
    $. == 1 or next;
    s/{^(.{26})(\d)(.{12})(\d)(.{6})(\d)/$1$c[$2]$3$c[$4]$5$c[$6]/;
' | ...

Somewhat more readable:

tac file | perl -pe '
    BEGIN { @c = qw/ { A B C D E F G H I /; }
    $. == 1 or next;
    s{^(.{26}) (\d)    (.{12}) (\d)   (.{6}) (\d)   }
     {   $1  . $c[$2]  . $3  . $c[$4] . $5 . $c[$6] }xe;
' | ...

As pointed out in xhienne's answer, your command did not work since $num and ${chars[$num]} were written inside single quotes '...' where bash does no expansion of variables. Use double quotes "..." instead.

In this answer, I just want to point out a trick to make the script more efficient. Right now you do sed -i ten times. That is, the file has to be read over and over again. You can do everything in a single sed command whose command string is dynamically generated:

#!/bin/bash

chars=( '{' {A..I} )
cmd=()
for num in "${!chars[@]}"; do
  for pos in 27 40 47; do
    cmd+=(-e "\$ s%^(.{$((pos-1))})$num%\\1${chars[num]}%")
  done
done
sed -Ei "${cmd[@]}" "$1"

Like your original script, this only works if chars is free of sed control characters like \, %, and &. If you need one of these chars escape them manually as in chars=('\\' '\%' '\&') or automatically with another sed command.

You program would be ok if your bash variables were substituted in your sed command, but the use of single quotes prevents this.

Use double quotes around your shell variables:

sed -i -E '$ s%^(.{26})'"$num"'%\1'"${chars[$num]}%"

Your input file needs to be 2 or more lines to test that only the last line changes, so lets start with this sample input:

$ cat file
210204154828000999990000056000000072328100000540000000
210204154828000999990000056000000072328100000540000000

and then given this script:

$ cat tst.awk
NR > 1 { print prev }
{ prev = $0 }
END {
    split("A B C D E F G H I",chars); chars[0]="{"
    split("27 40 47",poss)
    for (i in poss) {
        pos = poss[i]
        charIdx = substr(prev,pos,1)
        prev = substr(prev,1,pos-1) chars[charIdx] substr(prev,pos+1)
    }
    print prev
}

We get this output:

$ awk -f tst.awk file
210204154828000999990000056000000072328100000540000000
21020415482800099999000005F000000072328A000005D0000000

EDIT: FWIW here is the gawk equivalent of the perl script at the end of @GlennJackman's answer:

$ tac file |
    awk 'BEGIN { split("A B C D E F G H I",c); c[0]="}" }
    NR==1 && match($0,/^(.{26})([0-9])(.{12})([0-9])(.{6})([0-9])(.*)/,a) {
        $0 = a[1] c[a[2]] a[3] c[a[4]] a[5] c[a[6]] a[7]
    } 1' | tac
210204154828000999990000056000000072328100000540000000
21020415482800099999000005F000000072328A000005D0000000

Calling sed inside a loop is inefficient. Below is a bash script calling sed once to do the job without using any other external command (tested with GNU sed).

#!/bin/bash

digit='0123456789'
chars='{ABCDEFGHI'

# The below two assignments are not necessary if chars cannot contain
# the '\' and '/' characters
chars=${chars//\\/\\\\}
chars=${chars//\//\\\/}

first=27  # first position
middle=40 # middle position
last=47   # last position
f=$((first-1))        # number of characters before the first position
m=$((middle-first-1)) # number of characters between middle and first positions
l=$((last-middle-1))  # number of characters between last and middle positions

sed -i -E "
$ {
    h
    s/.{$f}(.).{$m}(.).{$l}(.).*/\1\2\3/
    y/$digit/$chars/
    G
    s/(.)(.)(.).(.{$f}).(.{$m}).(.{$l})./\4\1\5\2\6\3/
}
" "$1"

This is a straight sed version, changing positions 27, 40, and 47 of last line using a lookup table temporarily inserted at end of line; assuming those positions - and none of their replacements - are digits 0-9 only.

sed -E -e '
$!b
s/$/\n1A2B3C4D5E6F7G8H9I0{/
s/^(.{26})(.)(.[^\n]*)(\n.*\2(.).*)/\1\5\3\4/
s/^(.{39})(.)(.[^\n]*)(\n.*\2(.).*)/\1\5\3\4/
s/^(.{46})(.)(.[^\n]*)(\n.*\2(.).*)/\1\5\3/
' -- file

EDIT

Some additional explanations, as per comments:

  • $!b : unless on last line skip to next line
  • s/$/\n1A2B3C4D5E6F7G8H9I0{/ : append a newline followed by a lookup table to the end of line, interleaving lookups (digits 0-9) and their replacements (A-{)
  • the 3 s commands use grouping (…), multiple-character matching {…}, and back references \1 thru \5, to produce the desired output
  • each s captures the characters before position N as \1, char. at pos. N as \2, chars after pos. N up to newline as \3, lookup table as \4, and replacement as \5; using \2 to determine the lookup
  • last s strips the lookup table (no \4)

The -e option is often omitted before a short script string but I tend to use it when also using -- to mark the end of options. -E is a GNU option to enable extended regular expressions ERE - over the default basic ditto BRE. In this case ERE improves readability: you can use BRE here but then you'd have to escape all (){} - except those in the replacement sections of s commands.

The GNU -specific -i option (inline editing) can be convenient but AFAIK is slower than cmd -- file1 > file2 && mv -f -- file2 file1.

As for the italics in the code block: they have no special meaning here, I tried to get rid of them but SO's markdown engine isn't 100% compatble with the one I'm using (has to do with the *'s). editing-help says 'Markdown and HTML are ignored within a code block' but not wanting to make an escaping error I let it ride.

With perl how about:

perl -pe '
    if (eof) {
        for $i (27, 40, 47) {
            substr($_, $i - 1, 1) = (qw/ { A B C D E F G H I /)[substr($_, $i - 1, 1)];
        }
    }
' file

The substr() function can also be used as an l-value to be assigned.

{mawk/mawk2/gawk} -v VALMAPPING='27|6|40|1|47|4' 'BEGIN {

       RS = "^$"; ORS = FS = OFS = "\n";
       nx = split(VALMAPPING, arrMap, "|"); 

       for (x = 1; x < nx; x += 2) { src[arrMap[x]] = arrMap[x+1]; };

    } END { if ($(NF)=="") { NF--; };

       for (x=1; x < NF; x++) { print $(x) };
       FS = OFS = ""; $0 = $(NF);

       for (x in src) { 
          if (($(x) ~ /[0-9]/) && ($(src[x]) ~ /[0-9]/)) { $(x) = $(src[x]) }
       }; 

       print }' 

I presume you wanted the rows other than the last to be directly printed without being checked. so that's the first for( ) loop in the END section. Once done with that, change the FS to blank string to so every digit becomes its own field. Then replacements occur only when it exists.

Trade off is doing 3 for loops. Gain is no substr( ) gsub( ) or match( ) needed.

Related