WSL2 interop issue causes premature exit from read loop in shell script

user_16838766 0
Viewed 234

Summary

Using read loop that runs a windows executable in WSL shell script causes it to exit the loop after the first iteration.

Details

I've been quite baffled by what appears to be an interoperability issue with running windows executables from a shell script in WSL2. The following while loop should print 3 lines but it will only print "line 1". It has been tested on Ubuntu 20.04 in dash, bash, and zsh.

while read -r line; do
       powershell.exe /C "echo \"${line}\""
done << EOF
line 1
line 2
line 3
EOF

This issue also occurs when reading lines from a file instead of a heredoc even if that file has windows line endings. Note that if powershell were changed to /bin/bash or any other native executable this would print 3 lines. Also powershell could be replaced with any windows executable cmd.exe, explorer.exe, etc and it would still only run the first iteration. This appears to be a problem with read specifically since this loop will work fine.

for line in "line 1" "line 2" "line 3"
do
    powershell.exe /C "echo \"${line}\""
done

Work-around

Thanks to this post I have discovered a work around is to pipe through a dummy command: echo "" | cmd.exe /C "echo \"${line}\"". A note about this fix is that only piping seems to work. Redirecting the output or running it through another layer of bash does not: /bin/bash -c "cmd.exe /C \"echo ${line}\"". I am partially posting this for improved visibility for anyone having this issue in the future, but I am still curious if anyone has any insight as to why this issue exists (perhaps due to line endings?). Thank you!

windows-subsystem-for-linux shell bash
1 Answers

Short Answer:

A slightly-improved solution over echo "" | is to do a second-redirection from /dev/null. This avoids potential newline issues from the echo, but there are other solutions as well:

while read -r line; do
       powershell.exe /C "echo \"${line}\"" < /dev/null
done << EOF
line 1
line 2
line 3
EOF

Explanation:

Well, you already had a solution, but what you really wanted was the explanation.

Jetchisel and MarkPlotnick are on the right track in the comments. This appears to be the same root cause (and solution) as in this question about ssh. To replicate your example with ssh (assuming a key in ssh-agent so that no password prompt is generated):

while read -r line; do
  ssh hostname echo ${line}
  done << EOF
line 1
line 2
line 3
EOF

You will see the same results as with PowerShell -- Only "line 1" displays.

In both cases, the first line goes to the read statement, but the subsequent lines are stdin which are consumed by powershell.exe (or ssh) itself.

You can see this "proven" in PowerShell through a slight modification to your script:

while read -r line; do
  powershell.exe -c "echo \"--- ${line} ---\"; \$input"
  done << EOF
line 1
line 2
line 3
EOF

Results in:

--- line 1 ---
line 2
line 3

The follow-up question is, IMHO, why bash doesn't have this issue. The answer is that PowerShell seems to always consume whatever stdin is available at the time of invocation and adds it to the $input magic variable. Bash, on the other hand, does not consume the additional stdin until explicitly asked:

while read -r line; do
  bash -c "echo --- \"${line}\" ---; cat /dev/stdin"
  done << EOF
line 1
line 2
line 3
EOF

Generates the same results as the previous PowerShell example:

--- line 1 ---
line 2
line 3

Ultimately, the main solution with PowerShell is to force a second indirection which is consumed before your desired input. echo "" | can do this, but be careful:

while read -r line; do
  echo "" | powershell.exe -c "echo \"--- ${line} ---\"; \$input" 
  done << EOF
line 1
line 2
line 3
EOF

Results in:

--- line 1 ---

--- line 2 ---

--- line 3 ---

< /dev/null doesn't have this issue, but you could also handle it with echo -n "" | instead.

user_118109330
Related