Apply awk command to all files in a directory

user_15169065 0
Viewed 218

I'm new in awk and I would like to apply a simple awk command to all files in a directory and get the result for each file separately. The files are tab delimited and I just need to sum every value in column 11 and print the result separately for each file. I tried the following code but it does not work.

for i in *;
do
awk -F '\t' '{sum += $11} END {print sum} "$i"'
done

Thanks!

awk
2 Answers

You may use this gnu awk:

awk -F '\t' '{sum += $11} ENDFILE {print FILENAME ":", sum; sum=0}' *

ENDFILE block will run at the end of processing for each file where we are printing file name and sum.

If you don't have gnu awk then use this:

awk -F '\t' 'FNR==1 {if (sum) print fn ":", sum; sum=0; fn=FILENAME} 
{sum += $11} END {print fn ":", sum}' *
user_5482250

You seem to have a simple typo in your shell quoting; this should work:

for f in *; do awk -F '\t' '{sum += $11} END {print sum}' -- "$f"; done

(the -- is only necessary to protect against one of the filenames starting with a hyphen; a variable called i is usually integer by old Fortran convention so using one for a string is a little perverse); or alternatively

ls | while read -r f; do awk -F '\t' '{sum += $11} END {print sum}' -- "$f"; done

if you want avoid the risk of hitting the limit on the length of the command line (which is large but finite on current systems), though this whill break if your filenames have newlines in them (why?!..).

If you want to do everything in a single Awk invocation, you can keep track of files using FNR (number of record inside file) and FILENAME:

awk -F '\t' 'FNR==1 {if (f) print sum; sum = 0; f = FILENAME} END {print sum} {sum += $11}' -- *

or alternatively (may invoke awk one or more times)

ls | xargs awk -F '\t' 'FNR==1 {if (f) print sum; sum = 0; f = FILENAME} END {print sum} {sum += $11}' --

with the same caveats as above.

You can guard against newlines in filenames by giving up on POSIX and using GNUisms which separate items with NULs (prohibited in filenames) instead of newlines (not prohibited), but unless your script is to run in a truly hostile environment it’s probably not worth it.

user_18133490
Related