Program description: I have a very active server application running as a Windows Service used to do communication to IoT devices (more than thousands per minute). Some do UDP, some TCP, some HTTP, always establishing the connection from the server side. It is NOT a "UWP" application which I know suspend process this way. Host is a Windows Server 2019.
The project was previously targeting .NET 4.7.2 and used System.ServiceProcess.ServiceBase to run as a service. It was also doing all communication in sync mode. It did not have this issue.
I then did a massive upgrade and ported it the .NET5, using the Microsoft.Extensions.Hosting.Host class with UseWindowsService() and the MS DI container, and ported all communication code to async/await.
Now, most of the time the process shut down or crash, it get stuck in the Windows Task Manager in "suspended" state, as you can see in the screenshot below:
Only the first process is the active one, the rest are "ghosts". These ghosts aren't waiting for some network communication to timeout or something, they can remain there for weeks.
I can't kill the process. Doing right-click "end task" does nothing. Even in an administrator command prompt with an user with full administrative priviledge, I get this error:
taskkill /F /PID 9100
ERROR: The process with PID 9100 could not be terminated.
Reason: Access is denied.
The only way I can get rid of them is rebooting the server.
Fortunately, they don't waste resources, and they don't keep the files locked, so I can still update the program and restart it.
The same application running on a development machine or a test server don't have this issue, it only happens with high load on the production server.
The only thing I can think of is that I do not propagate the shutdown CancellationToken to all my async tasks, but they should all end up completing after a few seconds all the time anyway.
My question is: Is there any way I can track down what cause this symptoms? Is there any way I can diagnose what is keeping the process locked like this?