Unable to pipe infinite output of console app using PowerShell

Question

During development of a console app I noticed that I'm unable to pipe its output into itself in PowerShell.

I created a small repro (source below) that works like this:

PS> .\but-why.exe print # continuously prints a random number every 500 ms
1746112985
1700785785
331650882
...

PS> .\but-why.exe read # echoes stdin
foo                    # this was typed
read 'foo'
bar                    # this too
read 'bar'

PS> "foo","bar" | .\but-why.exe read
read 'foo'
read 'bar'

But when I try to feed the output of print into read nothing happens:

PS> .\but-why.exe print | .\but-why.exe read

Same when I redirect all outputs to the success stream:

PS> .\but-why.exe print *>&1 | .\but-why.exe read

However, when I use CMD everything works as expected:

CMD> but-why.exe print | but-why.exe read
read '317394436'
read '1828759797'
read '767777814'
...

Through debugging I found that the second instance .\but-why.exe read never seems to be started.

Maybe it's my rather old PS version?

PS> $host.Version

Major  Minor  Build  Revision
-----  -----  -----  --------
5      1      19041  610

Source of the console app (net5.0):

using System;
using System.Threading;

switch (args[0]) {
    case "print": Print(); break;
    case "read": Read(); break;
}

void Print() {
    var rng = new Random();
    while (true) {
        Console.WriteLine(rng.Next());
        Thread.Sleep(500);
    }
}

void Read() {
    string? text;
    while ((text = Console.ReadLine()) != null) {
        Console.WriteLine($"read '{text}'");
    }
}

user_45375 · Answer

You're seeing a design limitation in Windows PowerShell that has since been fixed in the cross-platform PowerShell [Core] 7+ edition:

When Windows PowerShell pipes data to an external program (which is then invariably text), it unexpectedly does not exhibit the usual streaming behavior.

That is, instead of passing the originating command's lines (stringified objects) on as they're being produced, Windows PowerShell tries to collect them all in memory first, before piping them to the external program.

In your case, because the first program never stops producing output, the Windows PowerShell engine never stops waiting for all output to be collected and therefore effectively hangs (until it eventually runs out of memory) - the target program is never even launched, because that only happens after collecting the output has finished.

Workarounds:

If feasible, switch to PowerShell [Core] 7+, where this limitation has been removed.
In Windows PowerShell, invoke your pipeline via cmd.exe, which does exhibit the expected streaming behavior, as you've observed.
```
# Workaround via cmd.exe
cmd /c '.\but-why.exe print | .\but-why.exe read'
```

Unable to pipe infinite output of console app using PowerShell

1 Answers