I'm hosting a rest api using .NET 6, hosted on AKS using the official image (mcr.microsoft.com/dotnet/aspnet:6.0).
During the day, when the traffic is crazy, the response avg response time for operation X rarely goes above 200ms, with the 99th percentile under 1sec. But during the middle of night (like 4am) there are some HUGE increase in response without any correlation, after all CPU is 100% clear, memory is basically empty, no concurrence, no nothing.
Operation X does not touch any kind of cache (whether in memory, redis, memchache, etc)
I've seen this behavior in the past. As it is pretty well documented, IIS used to put apps to sleep. This shouldn't happen here after all, this is hosted on Kestrel/Linux.
To check if this is the case here or not I created a simple app the loops a request for operation X every 2 sec and guess what? the response time kept at its peak performance! it performed great, have a look:
the blue line is avg response time and red line is response count. Look how on the previous days there's always a spike counter-balancing the request count...
This leads me into believing that there is some kind of "sleeping" going around... Other than keep a bot making phantom requests, what else can I do?