C# Starting Process leaks memory even though Killed and Disposed (on Linux)

Viewed 843

Note: according to tests (see Edit below), this occurs only on a Linux machine.

I have an ASP.NET Core Blazor application (using server-side hosting model) running on a Raspberry Pi. Part of the application's functionality is to dim/brighten screen based on when system was last interacted with. To do that, every 1 second or so I spawn a terminal child-process to run xprintidle, parse its output, and act accordingly.

I use DataDog for monitoring, and I am having a memory leak until the system crashes (it takes a few days to use up all memory, but it does occur eventually):
enter image description here

I have pinpointed that the following method is what leaks memory - if I skip calling it and use some constant timespan, the memory does not leak: I have following code to do so:

// note this code has some parts that aren't even needed - I was simply trying anything to solve this problem at this point
public async Task<TerminalResult> ExecuteAndWaitAsync(string command, bool asRoot, CancellationToken cancellationToken = default)
{
    using Process prc = CreateNewProcess(command, asRoot);
    // we need to redirect stdstreams to read them
    prc.StartInfo.RedirectStandardOutput = true;
    prc.StartInfo.RedirectStandardError = true;

    // start the process
    _log.LogTrace("Starting the process");
    using Task waitForExitTask = WaitForExitAsync(prc, cancellationToken);
    prc.Start();

    // read streams
    string[] streamResults = await Task.WhenAll(prc.StandardOutput.ReadToEndAsync(), prc.StandardError.ReadToEndAsync()).ConfigureAwait(false);

    // wait till it fully exits, but no longer than half a second
    // this prevents hanging when process has already finished, but takes long time to fully close
    await Task.WhenAny(waitForExitTask, Task.Delay(500, cancellationToken)).ConfigureAwait(false);
    // if process still didn't exit, force kill it
    if (!prc.HasExited)
        prc.Kill(true);  // doing it with a try-catch approach instead of HasExited check gives no difference
    return new TerminalResult(streamResults[0], streamResults[1]);
}

public Task<int> WaitForExitAsync(Process process, CancellationToken cancellationToken = default)
{
    TaskCompletionSource<int> tcs = new TaskCompletionSource<int>();
    IDisposable tokenRegistration = null;
    EventHandler callback = null;
    tokenRegistration = cancellationToken.Register(() =>
    {
        Unregister();
        tcs.TrySetCanceled(cancellationToken);
    });
    callback = (sender, args) =>
    {
        Unregister();
        tcs.TrySetResult(process.ExitCode);
    };
    process.Exited += callback;
    process.EnableRaisingEvents = true;

    void Unregister()
    {
        lock (tcs)
        {
            if (tokenRegistration == null)
                return;
            process.EnableRaisingEvents = false;
            process.Exited -= callback;
            tokenRegistration?.Dispose();
            tokenRegistration = null;
        }
    }

    return tcs.Task;
}

private Process CreateNewProcess(string command, bool asRoot)
{
    _log.LogDebug("Creating process: {Command}", command);
    Process prc = new Process();

    if (RuntimeInformation.IsOSPlatform(OSPlatform.Linux))
    {
        string escapedCommand = command.Replace("\"", "\\\"");
        // if as root, just sudo it
        if (asRoot)
            prc.StartInfo = new ProcessStartInfo("/bin/bash", $"-c \"sudo {escapedCommand}\"");
        // if not as root, we need to open it as current user
        // this may still run as root if the process is running as root
        else
            prc.StartInfo = new ProcessStartInfo("/bin/bash", $"-c \"{escapedCommand}\"");
    }
    else if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
    {
        prc.StartInfo = new ProcessStartInfo("CMD.exe", $"/C {command}");
        if (asRoot)
            prc.StartInfo.Verb = "runas";
    }
    else
        throw new PlatformNotSupportedException($"{nameof(ExecuteAndWaitAsync)} is only supported on Windows and Linux platforms.");

    prc.StartInfo.UseShellExecute = false;
    prc.StartInfo.CreateNoWindow = true;

    if (_log.IsEnabled(LogLevel.Trace))
    {
        _log.LogTrace("exec: {FileName} {Args}", prc.StartInfo.FileName, prc.StartInfo.Arguments);
        _log.LogTrace("exec: as root = {AsRoot}", asRoot);
    }

    return prc;
}

I spent a lot of time (over span of months - literally) trying various changes to solve this issue - WaitForExitAsync was overhauled a lot, tried different ways of disposing. I attempted to call GC.Collect() periodically. Also tried running the application with both Server and Workstation GC mode.

As I mentioned earlier, I am pretty sure it's this code that leaks - if I don't call ExecuteAndWaitAsync, there's no memory leak. The result class is also not stored by the caller - it simply parses a value and uses it right away:

public async Task<TimeSpan> GetSystemIdleTimeAsync(CancellationToken cancellationToken = default)
{
    ThrowIfNotLinux();

    const string prc = "xprintidle";
    TerminalResult result = await _terminal.ExecuteAndWaitAsync(prc, false, cancellationToken).ConfigureAwait(false);
    if (result.HasErrors || !int.TryParse(result.Output, out int idleMs))
        throw new InvalidOperationException($"{prc} returned invalid data.");
    return TimeSpan.FromMilliseconds(idleMs);
}

private static void ThrowIfNotLinux()
{
    if (!RuntimeInformation.IsOSPlatform(OSPlatform.Linux))
        throw new PlatformNotSupportedException($"{nameof(BacklightControl)} is only functional on Linux systems.");
}

Am I missing something? Is it Process class leaking, or the way I read output?

EDIT: As people in comments asked, I created minimum runnable code, basically fetching all relevant methods in a single class and execute in a loop. The code is available as a gist: https://gist.github.com/TehGM/c953b670ad8019b2b2be6af7b14807c2
I ran it both on my Windows machine and Raspberry Pi. On Windows, memory seemed stable, however on Raspberry Pi it was clearly leaking. I tried both xprintidle and ifconfig to make sure it's not an issue with xprintidle only. Tried both .NET Core 3.0 and .NET Core 3.1, and effect was largely the same. enter image description here

1 Answers

It is probably caused by a regression between .NET Core 2.2 and .NET Core 3.0 Apparently it will be fixed in version 3.1.7

Just starting the process causes the memory leak on linux, because of a non released handle

Issue has been tracked here https://github.com/dotnet/runtime/issues/36661

Related