I've got an ASP.NET Core 2 app that is leaking threads. How can I determine why the threads never die?
If I run the app for 10 minutes in production, IIS starts spitting out 502.3 Bad Gateway errors. I loaded up Process Explorer and looked at the app->Properties->Threads, and I see an ever-increasing thread count:
Nearly all the threads are stuck with the following native stack trace:
- ntoskrnl.exe!KeWaitForMutexObject+0x8d6
- ntoskrnl.exe!KeDelayExecutionThread+0x9a9
- ntoskrnl.exe!KeWaitForMutexObject+0x1cf
- ntoskrnl.exe!_misaligned_access+0x80d
- ntoskrnl.exe!SeAccessCheck+0x280
- ntoskrnl.exe!KeDelayExecutionThread+0xb49
- ntoskrnl.exe!KeWaitForMutexObject+0x1cf
- ntoskrnl.exe!KeWaitForMultipleObjects+0x2ce
- ntoskrnl.exe!ObWaitForMultipleObjects+0x29c
- ntoskrnl.exe!ObWaitForMultipleObjects+0x713
- ntoskrnl.exe!KeSaveStateForHibernate+0x2a33
- ntdll.dll!ZwWaitForMultipleObjects+0xa
- KERNELBASE.dll!GetProcessHeap+0x62 coreclr.dll+0x4d9c8
- coreclr.dll+0x4db61 coreclr.dll!MetaDataGetDispenser+0x55008
- System.Private.CoreLib.dll+0x58834c
These native stack traces don't tell me much, other than my web app is leaking threads. While Process Explorer can show me .NET stack traces for full .NET framework apps, it doesn't show me .NET stack traces for .NET Core apps.
So, how does one go about debugging such an issue?
I'm unable to repro this in my local dev environment. It appears to be related to high traffic or to my production server environment.
Things I've tried so far:
- Enabled stdout logs with my app: there's no clues about failure.
- Added a AppDomain.UnhandledException handler, writing to stdout. No clues.
- Looked at the Windows Event Logs. No clues.