Redirect both stdout and stderr to file, print stdout only

Question

I have a large amount of text coming in stdout and stderr; I would like to log all of it in a file (in the same order), and print only what comes from stdout in the console for further processing (like grep).

Any combination of > file or &> file, even with | or |& will permanently redirect the stream and I cannot pipe it afterwards:

my_command > output.log | grep something  # logs only stdout, prints only stderr
my_command &> output.log | grep something  # logs everything in correct order, prints nothing
my_command > output.log |& grep something  # logs everything in correct order, prints nothing
my_command &> output.log |& grep something  # logs everything in correct order, prints nothing

Any use of tee will either

print what comes from stderr then log everything that comes from stdout and print it out, so I lose the order of the text that comes in
log both in the correct order if I use |& tee but I lose control over the streams since now everything is in stdout.

example:

my_command | tee output.log | grep something  # logs only stdout, prints all of stderr then all of stdout
my_command |& tee output.log | grep something  # logs everything, prints everything to stdout
my_command | tee output.log 3>&1 1>&2 2>&3 | tee -a output.log | grep something  # logs only stdout, prints all of stderr then all of stdout

Now I'm all out of ideas.

This is what my test case looks like:

testFunction() { 
  echo "output"; 
  1>&2 echo "error"; 
  echo "output-2"; 
  1>&2 echo "error-2"; 
  echo "output-3"; 
  1>&2 echo "error-3";
}

I would like my console output to look like:

output
output-2
output-3

And my output.log file to look like:

output
error
output-2
error-2
output-3
error-3

For more details, I'm filtering the output of mvn clean install with grep to only keep minimal information in the terminal, but I also would like to have a full log somewhere in case I need to investigate a stack trace or something. The java test logs are sent to stderr so I choose to discard it in my console output.

user_8344060 · Answer

While not really a solution which uses redirects or anything of that order, you might want to use annotate-output for this.

Assume that script.sh contains your function, then you can do:

$ annotate-output ./script.sh                                                     
13:17:15 I: Started ./script.sh
13:17:15 O: output
13:17:15 E: error
13:17:15 E: error-2
13:17:15 O: output-2
13:17:15 E: error-3
13:17:15 O: output-3
13:17:15 I: Finished with exitcode 0

So now it is easy to reprocess that information and send it to the files you want:

$ annotate-output ./script.sh \
  | awk '{s=substr($0,13)}/ [OE]: /{print s> "logfile"}/ O: /{print s}'
output
output-2
output-3
$ cat logfile 
output
error
error-2
output-2
error-3
output-3

Or any other combination of tee sed cut ...

As per comment from @CharlesDuffy:

Since stdout and stderr are processed in parallel, it can happen that some lines received on stdout will show up before later-printed stderr lines (and vice-versa).

This is unfortunately very hard to fix with the current annotation strategy. A fix would involve switching to PTRACE'ing the process. Giving nice a (much) higher priority over the executed program could, however, cause this behaviour to show up less frequently.

_{source: man annotate-output}

Redirect both stdout and stderr to file, print stdout only

1 Answers