Useless use of cat?

user_1173542 0
Viewed 42353

This is probably in many FAQs - instead of using:

cat file | command

(which is called useless use of cat), correct way supposed to be:

command < file

In the 2nd, "correct" way - OS does not have to spawn an extra process.
Despite knowing that, I continued to use useless cat for 2 reasons.

  1. more aesthetic - I like when data moves uniformly only from left to right. And it easier to replace cat with something else (gzcat, echo, ...), add a 2nd file or insert new filter (pv, mbuffer, grep ...).

  2. I "felt" that it might be faster in some cases. Faster because there are 2 processes, 1st (cat) does the reading and the second does whatever. And they can run in parallel, which means sometimes faster execution.

Is my logic correct (for 2nd reason)?

cat shell command-line
9 Answers

I often use cat file | myprogram in examples. Sometime I am being accused of Useless use of cat (http://porkmail.org/era/unix/award.html). I disagree for the following reasons:

  • It is easy to understand what is going on.

    When reading a UNIX command you expect a command followed by arguments followed by redirection. It is possible to put the redirection anywhere but it is rarely seen - thus people will have a harder time reading the example. I believe

    cat foo | program1 -o option -b option | program2

    is easier to read than

    program1 -o option -b option < foo | program2

    If you move the redirection to the start you are confusing people who are not used to this syntax:

    < foo program1 -o option -b option | program2

    and examples should be easy to understand.

  • It is easy to change.

    If you know the program can read from cat, you can normally assume it can read the output from any program that outputs to STDOUT, and thus you can adapt it for your own needs and get predictable results.

  • It stresses that the program does not fail, if STDIN is not a file.

    It is not safe to assume that if program1 < foo works then cat foo | program1 will also work. However, it is safe to assume the opposite. This program works if STDIN is a file, but fails if the input is a pipe, because it uses seek:

    # works
< foo perl -e 'seek(STDIN,1,1) || die;print <STDIN>'

# fails
cat foo | perl -e 'seek(STDIN,1,1) || die;print <STDIN>'

Performance cost

There is a cost of doing the additional cat. To give an idea of how much I ran a few tests to simulate baseline (cat), low throughput (bzip2), medium throughput (gzip), and high throughput (grep).

cat $ISO | cat
< $ISO cat
cat $ISO | bzip2
< $ISO | bzip2
cat $ISO | gzip
< $ISO gzip
cat $ISO | grep no_such_string
< $ISO grep no_such_string

The tests were run on a low end system (0.6 GHz) and an ordinary laptop (2.2 GHz). They were run 10 times on each system and the best timing was chosen to mimic the optimal situation for each test. The $ISO was ubuntu-11.04-desktop-i386.iso. (Prettier tables here: http://oletange.blogspot.com/2013/10/useless-use-of-cat.html)

CPU                       0.6 GHz ARM
Command                   cat $ISO|                        <$ISO                            Diff                             Diff (pct)
Throughput \ Time (ms)    User       Sys        Real       User       Sys        Real       User       Sys        Real       User       Sys        Real
Baseline (cat)                     55      14453      33090         23       6937      33126         32       7516        -36        239        208         99
Low (bzip2)                   1945148      16094    1973754    1941727       5664    1959982       3420      10430      13772        100        284        100
Medium (gzip)                  413914      13383     431812     407016       5477     416760       6898       7906      15052        101        244        103
High (grep no_such_string)      80656      15133      99049      79180       4336      86885       1476      10797      12164        101        349        114

CPU                       Core i7 2.2 GHz
Command                   cat $ISO|           <$ISO             Diff          Diff (pct)
Throughput \ Time (ms)    User     Sys Real   User   Sys Real   User Sys Real User       Sys Real
Baseline (cat)                    0 356    215      1  84     88    0 272  127          0 423  244
Low (bzip2)                  136184 896 136765 136728 160 137131 -545 736 -366         99 560   99
Medium (gzip)                 26564 788  26791  26332 108  26492  232 680  298        100 729  101
High (grep no_such_string)      264 392    483    216  84    304   48 308  179        122 466  158

The results show that for low and medium throughput the cost is in the order of 1%. This is well within the uncertainty of the measurements, so in practice there is no difference.

For high throughput the difference is bigger and there is a clear difference between the two.

That leads to the conclusion: You should use < instead of cat | if:

  • the complexity of the processing is similar to a simple grep
  • performance matters more than readability.

Otherwise it does not matter whether you use < or cat |.

And thus you should only give a UUoC-award if and only if:

  • you can measure a significant difference in the performance (publish your measurements when you give the award)
  • performance matters more than readability.
user_3630280
Related