Unix decompress files parallel and store them

userX732 0
Viewed 19

I have a directory /user/test with 2000 compressed files. I want to check if any given file has 5 records then I have to store it in decompressed format.

I am able to do it serially but it is taking a lot of time to finish this job.

Serially I am doing as below:

for i in `find /user/test -iname "abc*.gz"`;
do
    lines=`zcat $i | wc -l`
    if [ $lines = 5 ]; then
        fname=`basename -s .$file_ext $i`
        echo "copying $fname to new path"
        `zcat $i > new_path/$fname`
        cnt=$((cnt+1))
    else
        echo "Ignoring file $i. Expecting 5 records. It has more or less records"
    fi
done

I want to do the same in parallel.

I tried exploring GNU parallel but am seeing an error. I tried below command

find /user/test -iname "abc*.gz" |
parallel 'zcat {} | awk 'NR == 5 {print $0}' < {}.txt'

Above command is not working throwing error.

unix parallel-processing
1 Answers

Untested:

doit() {
  zcat "$@" | awk 'NR == 5 {print $0}'
}
export -f doit
find /user/test -iname "abc*.gz" |
  parallel doit
Ole Tange0
Related