AWK taking very long to process file

sort file |
awk '{out=substr($0,1,17)".txt"} out != prev{close(prev); prev=out} {print > out}'

Performance improvements included:

1. By not referring to any field it lets awk not do field splitting
2. By sorting first and changing output file names only when the key part of the input changes, it lets awk only use 1 output file at a time instead of having to manage opening/closing potentially thousands of output files

And it's portable to all awks since it's not using gawk-specific extension like FIELDWIDTHS.

If the lines in each output file have to retain their original relative order after sorting then it'd be something like this (assuming no white space in the input just like in the example you provided):

awk '{print substr($0,1,17)".txt", NR, $0}' file |
sort -k1,1 -k2,2n |
awk '$1 != prev{close(prev); prev=$1} {print $3 > $1}'

After borrowing @dawg's script (perl -le 'for (1..120000) {print map { (q(A)..q(Z))[rand(26)] } 1 .. 17} ' | awk '{for (i=1; i<6; i++) printf ("%s%05i\n", $0, i)}' | awk 'BEGIN{srand();} {printf "%06d %s\n", rand()*1000000, $0;}'| sort -n | cut -c8- > /tmp/test/file - thanks!) to generate the same type of sample input file he has, here's the timings for the above:

$ time sort ../file | awk '{out=substr($0,1,17)".txt"} out != prev{close(prev); prev=out} {print > out}'

real    0m45.709s
user    0m15.124s
sys     0m34.090s

$ time awk '{print substr($0,1,17)".txt", NR, $0}' ../file | sort -k1,1 -k2,2n | awk '$1 != prev{close(prev); prev=$1} {print $3 > $1}'

real    0m49.190s
user    0m11.170s
sys     0m34.046s

and for @dawg's for comparison running on the same machine as the above with the same input ... I killed it after it had been running for 14+ minutes:

$ time awk -v  FIELDWIDTHS="17"  '{of=$1 ".txt"; if (of in seen){ print >>of } else {print >of; seen[of]; } close(of);}' ../file

real    14m23.473s
user    0m7.328s
sys     1m0.296s

I created a test file of this form:

% head file
SXXYTTLDCNKRTDIHE00004
QAMKKMCOUHJFSGFFA00001
XGHCCGLVASMIUMVHS00002
MICMHWQSJOKDVGJEO00005
AIDKSTWRVGNMQWCMQ00001
OZQDJAXYWTLXSKAUS00003
XBAUOLWLFVVQSBKKC00005
ULRVFNKZIOWBUGGVL00004
NIXDTLKKNBSUMITOA00003
WVEEALFWNCNLWRAYR00001
% wc -l file
  600000 file

ie, 120,000 different 17 letter prefixes to with 01 - 05 appended in random order.

If you want a version for yourself, here is that test script:

perl -le 'for (1..120000) {print map { (q(A)..q(Z))[rand(26)] } 1 .. 17} ' | awk '{for (i=1; i<6; i++) printf ("%s%05i\n", $0, i)}' | awk 'BEGIN{srand();} {printf "%06d %s\n", rand()*1000000, $0;}'| sort -n | cut -c8- > /tmp/test/file

If I run this:

% time awk -v  FIELDWIDTHS="17"  '{print > $1".txt"}' file

Well I gave up after about 15 minutes.

You can do this instead:

% time awk -v  FIELDWIDTHS="17"  '{of=$1 ".txt"; if (of in seen){ print >>of } else {print >of; seen[of]; } close(of);}' file

You asked about Perl, and here is a similar program in Perl that is quite fast:

perl -lne '$p=unpack("A17", $_); if ($seen{$p}) { open(fh, ">>", "$p.txt"); print fh $_;} else { open(fh, ">", "$p.txt"); $seen{$p}++; }close fh' file

Here is a little script that compares Ed's awk to these:

#!/bin/bash

# run this in a clean directory Luke!

perl -le 'for (1..12000) {print map { (q(A)..q(Z))[rand(26)] } 1 .. 17} ' 
| awk '{for (i=1; i<6; i++) printf ("%s%05i\n", $0, i)}' 
| awk 'BEGIN{srand();} {printf "%06d %s\n", rand()*1000000, $0;}'
| sort -n 
| cut -c8- > file.txt

wc -l file.txt

#awk -v  FIELDWIDTHS="17"  '{cnt[$1]++} END{for (e in cnt) print e, cnt[e]}' file
echo "abd awk"
time awk -v  FIELDWIDTHS="17"  '{of=$1 ".txt"; if (of in seen){ print >>of } else {print >of; seen[of]; } close(of);}' file.txt

echo "abd Perl"
time perl -lne '$p=unpack("A17", $_); if ($seen{$p}) { open(fh, ">>", "$p.txt"); print fh $_;} else { open(fh, ">", "$p.txt"); $seen{$p}++; }close fh' file.txt

echo "Ed 1"
time sort file.txt |
awk '{out=substr($0,1,17)".txt"} out != prev{close(prev); prev=out} {print > out}'

echo "Ed 2"
time sort file.txt | awk '{out=substr($0,1,17)".txt"} out != prev{close(prev); prev=out} {print > out}'

echo "Ed 3"
time awk '{print substr($0,1,17)".txt", NR, $0}' file.txt | sort -k1,1 -k2,2n | awk '$1 != prev{close(prev); prev=$1} {print $3 > $1}'

Which prints:

   60000 file.txt
abd awk

real    0m3.058s
user    0m0.329s
sys 0m2.658s
abd Perl

real    0m3.091s
user    0m0.332s
sys 0m2.600s
Ed 1

real    0m1.158s
user    0m0.174s
sys 0m0.992s
Ed 2

real    0m1.069s
user    0m0.175s
sys 0m0.932s
Ed 3

real    0m1.174s
user    0m0.275s
sys 0m0.946s