Process multiple files at once using awk and parallel

Viewed 563

I have more than a hundred part .csv files that contain data delimited by '|'. I need to add prefix on second column of each part file using awk and parallel.

I am doing single file at a time, but is taking hours. so wanted to go with parallel.

input1.csv

10|20

10|30

10|40

input2.csv

20|30

20|40

line1|10

output expected

input1.csv

10|P20

10|P30

10|P40

input2.csv

20|P30

20|P40

line1|P10

I am using this, and working when I do one file at a time, but I need something faster that will do multiple files using parallel

awk 'BEGIN{FS=OFS="|"} {$2 = "P" $2} 1' input1.csv > in.tmp && mv in.tmp input1.csv

awk 'BEGIN{FS=OFS="|"} {$2 = "P" $2} 1' input2.csv > in.tmp && mv in.tmp input2.csv

end finally merge all the input*.csv files into input

1 Answers

this does the trick for me

parallel "awk -F'|' '{print \$1\"|\"\"PO\"\$2}' {} > {}.tmp; mv {}.tmp {}" ::: input*.csv
Related