Log bash output (stderr, and stdout) to multiple files one at a time in one script

Question

Suppose you have a bash script and you want to print and also save the output (stderr, and stdout) to a log file. Based on this answer: https://stackoverflow.com/a/49514467/835098 by @cdarke, this is how you can do it.

#!/bin/bash

exec > >(tee my.log) 2>&1

echo "Hello"

But what if you have a script where different sections need to go to different log files? Let's say you want to separate the typical configure, make, make test output each into their individual log file? A naive approach could look like this (for the sake of simplicity configure and alike became echos here):

#!/bin/bash

# clear logs
rm -f configure.log make.log make_test.log

exec > >(tee configure.log) 2>&1
echo "configure"

exec > >(tee make.log) 2>&1
echo "make"

exec > >(tee make_test.log) 2>&1
echo "make test"

But when you execute this script, you'll notice that only the last output contains what it's supposed to contain:

$ tail *.log
==> configure.log <==
configure
make
make test

==> make.log <==
make
make test

==> make_test.log <==
make test

Also notice that each log file begins with the correct output. I thought about sticking with one log file and truncating it after copying an intermediate piece of it to the final destination. This script works but I wonder if it is any good:

#!/bin/bash

# clear logs
rm -f configure.log make.log make_test.log tmp.log

exec > >(tee tmp.log) 2>&1
echo "configure"
cp tmp.log configure.log && truncate -s 0 tmp.log

echo "make"
cp tmp.log make.log && truncate -s 0 tmp.log

echo "make test"
cp tmp.log make_test.log  && truncate -s 0 tmp.log

Here are the resulting log files:

$ tail *.log
==> configure.log <==
configure

==> make.log <==
make

==> make_test.log <==
make test

==> tmp.log <==

For example one downside of this approach is that a final log file will be available if the command succeeded. Actually, this is pretty bad and a good reason to find another solution.

user_5291015 · Answer

The reason is because, you are duplicating stdout file descriptor in each call to exec, causing your output from all the three echo calls to all the three files. i.e. after setting the initial exec, when you did

exec > >(tee make.log) 2>&1
echo "make"

Not only the recent echo'd statement goes to make.log, but also to configure.log, because your previous exec did set up write to that log file for contents from stdout. Now the first log file contains lines from both the echo'd statements. This is replicated again in your next case also.

To avoid this problem, set-up different file descriptors for each unique case, i.e.

exec 3> >(tee configure.log) 2>&1
echo "configure" >&3

exec 4> >(tee make.log) 2>&1
echo "make" >&4

exec 5> >(tee make_test.log) 2>&1
echo "make test" >&5

# Releasing the file descriptors back to shell
exec 3>&-
exec 4>&-
exec 5>&-

Or if you are using a bash version >4.1, you can let the shell assign you the file descriptors, so that you don't have to pick the numbers explicitly.

exec {fd1}> >(tee configure.log) 2>&1
echo "configure" >&"$fd1"

exec {fd2}> >(tee make.log) 2>&1
echo "make" >&"$fd2"

exec {fd3}> >(tee make_test.log) 2>&1
echo "make test" >&"$fd3"

When you do exec {fd1}>, bash automatically, populates fd1 variable with a file descriptor (usually greater than 10). Now you can use $fd1 like before in your call along with tee. As before, you can free up the used descriptors by closing them as

exec >&"$fd1"-
exec >&"$fd2"-
exec >&"$fd3"-

If you want the stderr also to go to the corresponding log files, change your re-direction 2>&1 to the corresponding file descriptors opened, i.e.

2>&3        
2>&"$fd1"

Log bash output (stderr, and stdout) to multiple files one at a time in one script

1 Answers