With this post, I try to help my team and hopefully other .NET teams around the world to help them troubleshoot slow requests in the Azure Web application running as an API for an Angular frontend. The goal is to create a small guideline together with the community. There is a lot of chaotic information out there, but each application is different. Maybe we can list up some checks based on the experience of other teams.
Context: We have an API running in a .NETCore 2.1.x Azure web application on:
- p3v2 app service plan.
- Separate S1 Web application for Angular frontend (out of scope of this post)
- Azure SQL Database Standard S6
- Azure Redis cache Standard 6GB
- Application Insights
Important Note: This post is about troubleshooting a n in general fast-running API application which is suddenly slowing down. So no specific API calls, but all requests are suddenly slow which means they cannot directly be linked to specific code.
The API application has an AVG load of 35k requests during 5 minutes (see application insights - requests) and 300-400 simultaneous users.
Usually the requests have an avg response of a few milliseconds.
When the application is running slow it can be caused by several factors. Please find below our first standard troubleshooting guideline:
- Check the Memory and CPU usage of the Azure service plan. Is it too high?
Ex: Once we had an issue of Out-of-memory exceptions caused by the Dotnet.exe app still running in 32bit. Make sure you deploy your .net application as 64bit if a lot of memory is needed - Check the Database DTU usage. If it's too high this can also cause slow API requests as they are waiting for the database to reply
- Check the load on your Redis cache instance and see if it's not too high.
Now, my question to the community: Are there other things that can slow down the application significantly but are not seen in the three bullets above?
The API was suddenly responding very slow yesterday, however, there were:
- no high CPU usage, nor high Memory usage of the app service plan
- no high DTU usage of the database
- no high usage of Redis cache
Maybe some examples I can think of:
- Can sudden application exceptions on one particular request slow down other API requests without influencing the three bullets above?
- Can too many requests, sent by one external client, slow down the application without influencing the three bullets above? If yes how can this be noticed?
- Other things that can be checked?