How to use Windows ConPTY API from a process whose output has been redirected?

Viewed 233

I am working on an application which creates a Windows pseudoconsole (using the new ConPTY API).

It starts a PowerShell session (for test), and shows everything ConPTY sends from that session via the pseudoconsole, and everything works really well. Until I redirect the parent program output to anywhere.

Here's my full code listing for reference.

#include <string>
#include <stdexcept>
#include <iostream>
#include <vector>

#include <Windows.h>

using namespace std::string_literals;

void throwLastError(int number)
{
    throw std::runtime_error("Error "s + std::to_string(number) + ": "s + std::to_string(GetLastError()));
}

STARTUPINFOEXW prepareStartupInformation(HPCON hPCon)
{
    STARTUPINFOEXW startupInfo{sizeof(STARTUPINFOEXW)};
    SIZE_T bytesRequired = 0;
    if (InitializeProcThreadAttributeList(nullptr, 1, 0, &bytesRequired))
        throw std::runtime_error("InitializeProcThreadAttributeList wasn't expected to succeed at that time.");

    const auto threadAttributeList = static_cast<LPPROC_THREAD_ATTRIBUTE_LIST>(calloc(bytesRequired, 1));
    startupInfo.lpAttributeList = threadAttributeList;
    
    if (!InitializeProcThreadAttributeList(threadAttributeList, 1, 0, &bytesRequired))
        throwLastError(4);

    if (!UpdateProcThreadAttribute(
            threadAttributeList,
            0,
            PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE,
            hPCon,
            sizeof(HPCON),
            nullptr,
            nullptr)
    )
        throwLastError(5);

    return startupInfo;
}

PROCESS_INFORMATION startProcess(STARTUPINFOEXW startupInfo, const std::wstring &commandLine)
{
    std::vector<wchar_t> cmdLineBuffer;
    for (auto x : commandLine)
        cmdLineBuffer.push_back(x);
    cmdLineBuffer.push_back(L'\0');
    
    PROCESS_INFORMATION processInfo{nullptr};
    if (!CreateProcessW(
        nullptr,
        cmdLineBuffer.data(),
        nullptr,
        nullptr,
        FALSE,
        EXTENDED_STARTUPINFO_PRESENT,
        nullptr,
        nullptr,
        &startupInfo.StartupInfo,
        &processInfo)
    )
        throwLastError(6);

    return processInfo;
}

int main()
{
    auto h = GetStdHandle(STD_OUTPUT_HANDLE);
    auto type = GetFileType(h);
    std::cout << "STD_OUTPUT_HANDLE " << h << " " << type << std::endl;
    
    try
    {
        HANDLE inPipeRead = nullptr, inPipeWrite = nullptr;
        if (!CreatePipe(&inPipeRead, &inPipeWrite, nullptr, 0))
            throwLastError(1);
    
        HANDLE outPipeRead = nullptr, outPipeWrite = nullptr;
        if (!CreatePipe(&outPipeRead, &outPipeWrite, nullptr, 0))
            throwLastError(2);
    
        HPCON hPCon = nullptr;
        if (CreatePseudoConsole(COORD { 80, 25 }, inPipeRead, outPipeWrite, 0, &hPCon) != S_OK)
            throwLastError(3);

        CloseHandle(inPipeRead);
        CloseHandle(outPipeWrite);
    
        auto startupInfo = prepareStartupInformation(hPCon);

        auto processInfo = startProcess(startupInfo, L"powershell.exe");
    
        std::cout << "PID: " << processInfo.dwProcessId << "\nTID: " << processInfo.dwThreadId << "\n";

        char buffer[1024];
        DWORD readBytes = 0;
        while (ReadFile(
            outPipeRead,
            buffer,
            sizeof buffer - 1,
            &readBytes,
            nullptr
        ))
        {
            buffer[readBytes] = 0;
            std::cout << "Read bytes: " << readBytes << "\n";
            std::cout << buffer << "\n";
        }
    }
    catch (const std::runtime_error &x)
    {
        std::cerr << x.what() << "\n";
    }
}

So, if I start this program from terminal or from an IDE, it works (and tells how much data it has read from the PTY pipe). But if I start it with output redirected (e.g. myprogram.exe > C:\Temp\somefile.txt, or even myprogram.exe | out-host in pwsh), then it stops working: the PowerShell then somehow inherits the stdin and stdout and doesn't write anything to the PTY pipe.

For diagnostics, I've added output of GetFileType(GetStdHandle(STD_OUTPUT_HANDLE)), which helps to detect if the output has been redirected (since it's not always obvious in practice). For pristine stdout, it should write 2, and other numbers for other output modes.

How can I overcome this issue? Is it possible for my program to work even if its own stdout/stderr were redirected somewhere? I thought one of the points of PTY was to create a separate isolated environment, which won't interfere with the parent one.

1 Answers

This got answered dy Dustin L. Howett on Microsoft Terminal issue tracker:

A-ha! We did do some work to improve handle inheritance, but only if the inherited handles were console handles (so: the spawned application would connect to its parent's stdin/stdout even if it was supposed to be using a pseudoconsole)

    //
    // If we are establishing a Pseudoterminal Connection, close
    // any in/out/err handles that may have been copied from the parent.
    // We will re-make new connections to the given PTY instead of
    // using the automatically inherited handles.
    //
        
    if ((ProcessParameters->ConsoleFlags & CONSOLE_USING_PTY_REFERENCE) != 0)
    {
        ConsoleCloseIfConsoleHandle(&ProcessParameters->StandardInput);
        ConsoleCloseIfConsoleHandle(&ProcessParameters->StandardOutput);
        ConsoleCloseIfConsoleHandle(&ProcessParameters->StandardError);
    }

We didn't do the same for all other inherited handles, so... I'm going to file a bug on the docs to make sure we document the need for STARTF_USESTDHANDLES.

So, for such cases, it's required to pass STARTF_USESTDHANDLES in startupInfo.StartupInfo.dwFlags.

Related