Can't successfully call any external shell script in pipe with set -eo pipefail

user_14007501 0
Viewed 407

Suppose the following test-pipefail.sh batch:

#!/usr/bin/env bash
set -eo pipefail
./echoer.sh | head -n1 >/dev/null
echo "Might work with pipefail"
for i in {1..100} ; do
    ./echoer.sh | head -n1 >/dev/null
done
echo "Stable work with pipefail"

With echoer.sh contents:

#!/usr/bin/env bash
echo 'head (GNU coreutils) 8.30'
echo 'GNU bash, version 5.0.16(1)-release (x86_64-pc-linux-gnu)'
exit 0

Expected results of ./test-pipefail.sh:

Might work with pipefail
Stable work with pipefail

Actual behaviour:

Might work with pipefail

or (at random) with no output.

Writer program in pipe never fails if I'm using any binary utility instead of echoer.sh, but it always doesn't work (cause pipefail-script to exit) if writer is shell script (for example, ldd from glibc binary package). Replacing execution (./echoer.sh) in test-pipefail.sh with sourcing (. echoer.sh) leads to increased probability of successful execution, i.e. sometimes I'm getting

Stable work with pipefail

in test-pipefail.sh output.

head always returns success in such pipes. Removing second echo in echoer.sh leads to successfull execution with both sourcing and execution in separate shell.

built-in pipe shell linux bash
2 Answers

Let's reduce the problem down to the basics. Consider:

$ (set -o pipefail; cat /dev/zero | head -c10; declare -p PIPESTATUS)
declare -a PIPESTATUS=([0]="141" [1]="0")

What happens is that, when head has had its fill, it finishes, closing the pipe. The preceding command, cat in this case, gets a SIGPIPE (13) signal. Consequently, it sets its exit code to 128+13=141 to indicate failure.

So, the issue is whether the first process is still running when the second process, head, finishes. Sometimes, your echoer.sh is faster than head and sometimes it is slower.

Since we are multitasking two processes, the timing will always be variable.

Sourcing vs executing

Replacing execution (./echoer.sh) in test-pipefail.sh with sourcing (. echoer.sh) leads to increased probability of successful execution

Sourcing eliminates the need to initialize a new shell which likely leads to faster execution and therefore greater likelihood of finishing before head.

Binary programs

Writer program in pipe never fails if I'm using any binary utility instead of echoer.sh

My cat example above shows the opposite. That is because the cat /dev/zero program will never finish, thus ensuring that it will eventually receive the SIGPIPE.

user_30303050

I think the result depends on how fast the writer finishes. If it finishes very quickly then it has no chance to be sent with SIGPIPE.

For example:

[STEP 119] # hexdump -n100 /dev/urandom | head -n1; echo '$?'=$?
0000000 eea2 36e7 24d8 15de 620c e258 f9d8 f138
$?=0
[STEP 120] # hexdump -n1000 /dev/urandom | head -n1; echo '$?'=$?
0000000 cf81 dd51 1594 88b2 c9c1 6c8a bbbd c80f
$?=0
[STEP 121] # hexdump -n1000 /dev/urandom | head -n1; echo '$?'=$?
0000000 ef2d b2d3 1024 af9f ee1e a5e6 5528 699e
$?=0
[STEP 122] # hexdump -n2000 /dev/urandom | head -n1; echo '$?'=$?
0000000 d9f7 6a0d 633b c1f7 8928 cef8 3ea9 6f5a
$?=141
[STEP 123] # hexdump -n2000 /dev/urandom | head -n1; echo '$?'=$?
0000000 c044 dbb0 c227 1836 9fb5 f03b b2d1 0605
$?=141
[STEP 124] #
user_9000780
Related