specific alphabet repetition pattern match using awk

Viewed 209

Suppose I have:

ttb_5_x1
tgg_5_x2
ttb_5_x5
tcc_8_x8
ccr_5_x4

I try to filter the lines based on the t alphabet in the begining. So lines starts with tt should be saved in a seperate file. Same for the lines start with t

Desired output should be:

ttb_5_x1
ttb_5_x5

and

tgg_5_x2
tcc_8_x8

I tried awk '/^t/{print}' or awk '/^tt/{print}' but it is not sensitive to 1 or 2 times repetition of t alphabet.

5 Answers

This can be done in a simpler and shorter awk in a single condition { action } block:

awk 'match($0,/^t+/) {print > ("outfile" RLENGTH)}' file

Then check output:

cat outfile1
tgg_5_x2
tcc_8_x8

cat outfile2
ttb_5_x1
ttb_5_x5

You may use

awk '{if($0 ~ /^tt/) {print > "outfile1"} else if ($0 ~ /^t/) {print > "outfile2"}}' file

If a line starts with tt (see if($0 ~ /^tt/)), it will be saved in outfile1, else, if the line starts with t (see if ($0 ~ /^t/)), it will be saved in outfile2. The order must be exactly this: first, check for tt, then for a single t.

outfile1 contents:

ttb_5_x1
ttb_5_x5

outfile2 contents:

tgg_5_x2
tcc_8_x8

Using awk:

$ cat file
ttb_5_x1
tgg_5_x2
abc_d_ef
ttb_5_x5
tcc_8_x8
$ awk '/^tt/ { print > "file1" } /^t[^t]/ { print > "file2" }' file
$ cat file1
ttb_5_x1
ttb_5_x5
$ cat file2
tgg_5_x2
tcc_8_x8
$
awk '/^t/{print > (/^tt/ ? "file2" : "file1")}' file

or maybe slightly faster since the above always does 2 regexp comparisons per line while the following only does 1 comparison for lines that start with tt:

awk '/^tt/{print > "file2"; next} /^t/{print > "file1"}' file

Here are the third-run timings for executing the above 2 scripts and all of the other solutions posted so far on a 10 million line file produced by duplicating the OPs input 2 million times using this script:

awk '{for (i=1;i<=2000000;i++) print}' file5 > file10m

ordered by speed of execution (slowest to fastest):

Anubahva's awk:

$ time awk 'match($0,/^t+/) {print > ("anubhava_outfile" RLENGTH)}' file10m

real    0m8.184s
user    0m8.042s
sys 0m0.124s

sexcpect's awk:

$ time awk '/^tt/ { print > "sexpect_file1" } /^t[^t]/ { print > "sexpect_file2" }' file10m

real    0m6.625s
user    0m6.480s
sys 0m0.128s

mathguy's 2 greps:

$ time { grep '^tt' file10m > grep_outfile1; grep -E '^t([^t]|$)' file10m > grep_outfile2; }

real    0m5.938s
user    0m5.812s
sys 0m0.111s

Eds awk 1:

$ time awk '/^t/{print > (/^tt/ ? "awk1_file2" : "awk1_file1")}' file10m

real    0m2.656s
user    0m2.535s
sys 0m0.110s

Eds awk 2:

$ time awk '/^tt/{print > "awk2_file2"; next} /^t/{print > "awk2_file1"}' file10m

real    0m2.604s
user    0m2.490s
sys 0m0.106s

Wiktor's awk:

$ time awk '{if($0 ~ /^tt/) {print > "wiktor_outfile1"} else if ($0 ~ /^t/) {print > "wiktor_outfile2"}}' file10m

real    0m2.421s
user    0m2.315s
sys 0m0.096s

So, no surprises there.

The above were run on a MacBook Pro with 2.2Ghz 6-Core Intel Core i7 processor with these versions of the tools:

$ grep --version
grep (BSD grep) 2.5.1-FreeBSD

$ awk --version
GNU Awk 5.0.1, API: 2.0 (GNU MPFR 4.0.2, GNU MP 6.1.2)

BSD awk was slower than GNU awk but BSD grep was faster than GNU grep so I used the timings from that fastest tools versions I have on my machine since that's what you'd use if you cared about performance.

I would do it in two passes of grep. It should be quite a bit faster than the awk solutions, even though it reads the input file twice. There are a few reasons for that: awk is a big machine, with much more overhead than grep. Also, writing alternatively to one file or the other means opening and closing those files repeatedly. With the solution below, we only write to each file once. (Unless I misunderstand how awk writes to those files - perhaps it opens them and then it keeps them open and available until awk completes?)

$ grep    '^tt'        file > outfile1
$ grep -E '^t([^t]|$)' file > outfile2

EDIT

I did some testing. I generated a file with 1.6 million lines, each consisting of exactly one string of 20 characters - a mix of lower-case ASCII letters and digits. Then I ran my solution (using two grep commands) against the accepted answer, using awk. Both solutions generated the same two output files - there are 960 strings that begin with tt and 46,160 strings that begin with a single t (followed by a character that is not t).

The "two times grep" solution takes 0.058 seconds on my machine. The awk solution takes 0.402 seconds. That's seven times longer.

Further comparisons: anubhava's awk answer (which puts strings which begin with three t in a different file, outfile3) takes 0.278 seconds, 4.8 times longer than the "two times grep" solution.

Ed Morton's solutions take 0.191 seconds (first solution - this is still 3.3 times longer than the two-grep solution) and 0.305 seconds (second solution, which is therefore slower, not faster, than the first).

For anyone who wants to repeat the tests: I used this command

$ head -c 4000000 /dev/random | tr -dc a-z0-9 | fold -w 20 | head -n 20000 > myfile

to generate 20,000 rows. Then I concatenated the file to itself (and then the result to itself) a few times to grow it to 1.6 million rows. About 1/36 of the rows begin with t, and of those, about 1/36 begin with two t.

Related