Getting FILENAME in awk for multiple compressed files

user_14286490 0
Viewed 96

I am trying to run a command on multiple files that would print the row if the first column is equal to “aaa” or is equal to “bbb” and add a column before that with the name of the file.

Sample file (sample.tsv.gz):

aaa, 111, 222
bbb, 333, 444
ccc, 555, 666

I want the output to be:

sample.tsv.gz, aaa, 111, 222
sample.tsv.gz, bbb, 333, 444

My command is:

zcat *.tsv.gz | awk '{if ($1==“aaa”||$1==“bbb”) {print FILENAME (NF?",":"") $0}}'

The output I get is:

-, aaa, 111, 222
-, bbb, 333, 444

How do I fix this?

filenames awk shell bash
3 Answers

Your command is parsing stdin provided by the output of your previous command, so filename is not available. One way to deal with it is this:

for f in *.tsv.gz; do
    zcat "$f" | awk -F, -v f="$f" '$1=="aaa" || $1=="bbb"{print f (NF?", ":"") $0}'
done
user_75896360

Could you please try following, find + xargs + awk solution.

find *.gz -print0 | 
xargs -0 -I fname sh -c 'gzip -dc fname | awk -v file="fname" -f prog.awk'

Where prog.awk program file should have following contents, this takes care of OP's condition of checking if first line's 1st is aaa or 2nd field is bbb.

cat prog.awk
BEGIN{FS=OFS=","}
{
 if($1=="aaa" || $1=="bbb"){
   print file,$0
 }
}

Testing of above solution:

  1. Let's create a test Input_file with name file as follows:
  2. Create its gzip -k Input_file
  3. Check if we could read its contents or not by doing gzip -dc file1.gz
  4. Finally run command mentioned above to check if its working fine or not.
##1st step
cat file1
aaa,bbb,ccc

##2nd step
gzip -k file1

##3rd step
gzip -dc file1.gz
aaa,bbb,ccc

##Execute solution:
find *.gz -print0 |
> xargs -0 -I fname sh -c 'gzip -dc fname | awk -v file="fname" -f prog.awk'
 Input_file.gz,aaa,bbb,ccc
user_58665800

Another awk (v.3) that runs the zcat within:

awk '
BEGIN {
    FS=OFS=", "                          # delimiter in the sample data

    for(i=1;i<ARGC;i++) {                # iterating the argument files
        zcat="zcat " ARGV[i]             # zcat command
        while((zcat | getline)>0)        # process while data left
            if ($1=="aaa"||$1=="bbb") {
                print ARGV[i], $0
            }
    }
}' *.gz

Output with two copies of your data:

bar.gz, aaa, 111, 222
bar.gz, bbb, 333, 444
foo.gz, aaa, 111, 222
foo.gz, bbb, 333, 444
user_41623560
Related