Since a few weeks, I see very long response times from my webapp. It's a .NET app hosted in an App Service on Azure. I'm talking >20 seconds for a simple request.
In the following request, the webapp gets a request, it checks if the authentication token is valid, then it queries the database and run some simple code to return some json back.
To use this .NET backend service, user needs to be authenticated with Azure AD, we're using a JWT token for that. Every time the webserver receives some request, it directly asks the Azure AD if the user is authenticated. There is almost no code touched before that Azure AD request is fired.
Normally everything above (check auth, do db query, parse request to json) should cost <500ms.
But now, sometimes, we can see this in Application Insights:
What I see is that the webserver 'hangs' for around 12 seconds before it asks Azure AD if the token is valid!
Besides, everything what's happening after that, costs also way too much time. I also don't understand the gap between the two database requests. The second request could never take >1000ms, but in the screenshot it does.
We got a lot of those slow responses around the same timeframe. An hour later, everything is back and snappy (then the striped bars from screenshot starts right after each other, like its supposed to), and another hour later, its back slow again.
I have no idea where I can start my search. I hope you guys could give me some search directions and/or ideas why this could happen.
I already checked the CPU load and memory load of the server. It is sometimes high (peaks >80%) but I don't see clear bottlenecks around 100%. Also, that load is relatively constant and shares no peaks with the slow moments, so I don't think the problem is there.
Edit: I used the Web Service Diagnostic to check out if that can give me some directions. Here I see the following thing:
HandlerChanged From(ExtensionlessUrlHandler-Integrated-4.0) to (System.Web.Http.WebHost.HttpControllerHandler) and ManagedPipelineHandler.
Does anyone know what that is and how I can debug that?