Troubleshoot threadpool starvation under heavy load

Armand 0
Viewed 69

We are experiencing issues with high load on our dotnet-core (3.1) application.

Beyond a certain amount of connection (virtual users), we encouter a bottleneck, the server is starved and we get request timeout but the process doesn't crash (no kestrel logs). We are using K6 to benchmark our app. For now the load test only performs GET requests on the login page which trigger one basic SQL request on a small dataset (no join, etc).

We used Visual Studio 2019 Perfomance Profiler tool and perfview to investigate the issue, but none of these tools helped us to identify the portion of code that caused this bottleneck.

I found this article about ThreadPool starvation : https://docs.microsoft.com/fr-fr/archive/blogs/vancem/diagnosing-net-core-threadpool-starvation-with-perfview-why-my-service-is-not-saturating-all-cores-or-seems-to-stall When we tweak the minimum ThreadPool with arbitrary values as the example after, we've got a huge improvement in performance (not on the graph). This seems like a stop gap, how bad is it to use it ?

System.Threading.ThreadPool.SetMinThreads(200, 200);

benchmarks that show the starvation Explanation : 2C_2G/100.csv => 2 cores, 2Go RAM, 100 virtual users

Environment:

  • nginx as reverse proxy
  • K6 as benchmark tool
  • dotnet-core 3.1 (with EntityFramework)
  • operating system : Ubuntu 20.04
  • mariadb as database
kestrel mariadb .net-core nginx c#
1 Answers

You're executing long-running code while on the thread pool.

Here's a way to do that with Task.Run:

public async Task<byte> CalculateChecksumAsync(Stream stream) => await Task.Run(() =>
{
    int i;
    byte checksum = 0;
    while ((i = stream.ReadByte()) >= 0)
    {
        checksum += (byte)i;
    }
    return checksum;
});

To the casual observer that looks like completely async code because there's async/await and Task everywhere.

But in fact that will tie up a thread pool thread for as long as it takes to read the stream (which depends not just on how much data comes through, but the bandwidth of the stream as well).

When the thread pool is starved then there's a one-second delay before the thread pool will spawn a new thread. That means that subsequent calls to Task.Run will have their work delayed for that long even if your CPU is sitting idle.

Alternatives:

  • Use async methods instead of synchronous methods where possible (e.g. Stream.ReadAsync), especially when you're on the thread pool
  • Spawn long-running tasks for long-running code: 
    public async Task<byte> CalculateChecksumAsync(Stream stream) => await Task.Factory.StartNew(() =>
{
    int i;
    byte checksum = 0;
    while ((i = stream.ReadByte()) >= 0)
    {
        checksum += (byte)i;
    }
    return checksum;
},
TaskCreationOptions.LongRunning);

The TaskCreationOptions.LongRunning flag tells C# that you want a new thread spawned immediately just for your work.

Matt Thomas0
Related