echo without trimming the space in awk command

user_9176914 0
Viewed 314

I have a file consisting of multiple rows like this 

10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GRCHEMBUR      MHIN|0000000010000.00|6761857316|508998|6011|GL

I have to split and replace the column 11 into 4 different columns using the count of character.

This is the 11th column containing extra spaces also.

SHOP NO.5,6,7 RUNWAL GRCHEMBUR      MHIN

This is I have done 

ls *.txt *.TXT| while read line
do
subName="$(cut -d'.' -f1 <<<"$line")"
 awk -F"|"   '{ "echo  -n "$11" | cut -c1-23" | getline ton;
             "echo -n  "$11" | cut -c24-36" | getline city;
             "echo -n  "$11" | cut -c37-38" | getline state;
             "echo -n  "$11" | cut -c39-40" | getline country;
$11=ton"|"city"|"state"|"country; print $0

}' OFS="|" $line > $subName$output

done

But while doing echo of 11th column, its trimming the extra spaces which leads to mismatch in count of character. Is there any way to echo without trimming spaces ?

Actual output

10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GR|CHEMBUR MHIN|||0000000010000.00|6761857316|508998|6011|GL

Expected Output

10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GR|CHEMBUR|MH|IN|0000000010000.00|6761857316|508998|6011|GL
linux bash
2 Answers

The least annoying way to code this that I've found so far is:

perl -F'\|' -lane '$F[10] = join "|", unpack "a23 A13 a2 a2", $F[10]; print join "|", @F'

It's fairly straightforward:

  • Iterate over lines of input; split each line on | and put the fields in @F.
  • For the 11th field ($F[10]), split it into fixed-width subfields using unpack (and trim trailing spaces from the second field (A instead of a)).
  • Reassemble subfields by joining with |.
  • Reassemble the whole line by joining with | and printing it.

I haven't benchmarked it in any way, but it's likely much faster than the original code that spawns multiple shell and cut processes per input line because it's all done in one process.

A complete solution would wrap it in a shell loop:

for file in *.txt *.TXT; do
    outfile="${file%.*}$output"
    perl -F'\|' -lane '...' "$file" > "$outfile"
done

Or if you don't need to trim the .txt part (and you don't have too many files to fit on the command line):

perl -i.out -F'\|' -lane '...' *.txt *.TXT

This simply places the output for each input file foo.txt in foo.txt.out.

user_18486540

A pure-bash implementation of all this logic 

#!/usr/bin/env bash
shopt -s nocaseglob extglob
for f in *.txt; do
  subName=${f%.*}
  while IFS='|' read -r -a fields; do
    location=${fields[10]}
    ton=${location:0:23}; ton=${ton%%+([[:space:]])}
    city=${location:23:12}; city=${city%%+([[:space:]])}
    state=${location:36:2}
    country=${location:38:2}
    fields[10]="$ton|$city|$state|$country"
    printf -v out '%s|' "${fields[@]}"
    printf '%s\n' "${out:0:$(( ${#out} - 1 ))}"
  done <"$f" >"$subName.out"
done

It's slower (if I did this well, by about a factor of 10) than pure awk would be, but much faster than the awk/shell combination proposed in the question.

Going into the constructs used:

  • All the ${varname%...} and related constructs are parameter expansion. The specific ${varname%pattern} construct removes the shortest possible match for pattern from the value in varname, or the longest match if % is replaced with %%.
  • Using extglob enables extended globbing syntax, such as +([[:space:]]), which is equivalent to the regex syntax [[:space:]]+.
user_141220
Related