We have a large legacy with asp.net system, we have started using some async methods from an infrastructure library that we cannot change. The system doesn't use tasks in most places but the infrastructure exposes only async methods.
In the code we use the following pattern to use the async methods:
Task.Run(() => Foo()).Result
We use the Task.Run to prevent a deadlock if somewhere in the code someone didn't use ConfigureAwait(false), There are a lot of places that someone could have missed and it has happened before. And we use Task.Result to integrate it with the existing sync code base.
After experiencing heavy load We've noticed that that we are getting timeouts but the servers are not doing any work(Low CPU), we've found out that when there are a lot of calls to the server and the thread pool reaches the maximal number of threads the threads reach a deadlock in the Task.Result as it blocks the thread until the task is finished but the task can't run as there are no thread pool threads available to run it.
The best solution would be to change the code to work async all the way, but that is not an option right now. Also removing the Task.Run could work but it's too risky as there is not enough test coverage to know that we will not cause new deadlocks in untested flows.
I've tried to implement a new task scheduler that will not use the thread pool but a different set of threads to run the Foo task, but internal task are being executed on the default task scheduler that I do not want to replace.
Any ideas how this can be resolved without a huge change to the code base?
This is a small sample app that reproduces the issue, using only 10 threads and not the real limit. In the sample Foo will never be called.
class Program
{
static void Main(string[] args)
{
ThreadPool.SetMaxThreads(10, 10);
ThreadPool.SetMinThreads(10, 10);
for (int i = 0; i < 10; i++)
{
ThreadPool.QueueUserWorkItem(CallBack);
}
Console.ReadKey();
}
private static void CallBack(object state)
{
Thread.Sleep(1000);
var result = Task.Run(() => Foo()).Result;
}
public static async Task<string> Foo()
{
await Task.Delay(100);
return "";
}
}