I have 3000+ messages per minute being delivered to an Azure Service Bus Queue. In AKS I had 10 instances of a pod that implements a dotnet core 5 Web Job that triggers off of a ServiceBusTrigger (Microsoft.Azure.WebJobs.Extensions.ServiceBus, Version=5.0.0-beta.2).
public async Task RunAsync([ServiceBusTrigger("minutedata", Connection = "AzureWebJobsServiceBusConnection")] ServiceBusReceivedMessage message, ServiceBusMessageActions messageActions, ILogger log)
Up until recently my AKS cluster had four nodes running around 25% CPU capacity. This was running 1.18.2 of Kubernetes. This configuration was handling the message load OK - all new messages were being processed within 1 minute of receipt, before the next messages were delivered. It was also running 40+ other pods OK at this CPU level.
After an upgrade to Kubernetes 1.20.5 I noticed that the queue length started to grow rapidly indicating that the messages were no longer being processed within 1 minute by the 10 processor instances. I also noticed that the average CPU % for each node was over 90%.
I scrambled to try to fix the situation so at least the queue length would settle and the backlog be processed. This included adding more nodes (up to 8 in total) and more processor instances (up to 30).
Once the queue was stabilized I started trying to figure out what was going on and to draw back the nodes and processor instances. I managed to get back to 5 nodes and 20 processor instances in order to stay on top of the 3000+ messages coming in per minute.
However, the nodes are now running at ~60% CPU average, and I'm using double the number of processor instances, just to keep up with a messaging throughput that was previously being handled by 4 nodes (running at 25%) and 10 processors.
I have re-written the processor (Web Job) to use the latest libraries (which is why I'm using Microsoft.Azure.WebJobs.Extensions.ServiceBus, Version=5.0.0-beta.2, I would never normally be using a beta library in production), and this has had no significant impact. I even creating a new Service Bus namespace and moved over to a new queue in case somehow the original one had been corrupted. Again, no impact.
I'm beginning to believe that my focus should be on what changed from version 1.18.2 to 1.20.5 of Kubernetes, but I'm not sure where to start. I read somewhere that Kubernetes deprecated Docker after 1.20, but I'm not sure if I need to do anything differently given that I'm using AKS (surely AKS will be automatically configured to run Docker containers running dotnet core apps).
Another observation is that the messaging throughput drops off every 15 minutes and recovers within 2 minutes.
I was wondering if there's a default setting in Kubernetes that is now affecting the number of connections that can be made from each node that might be impacting the processors ability to receive and process messages from the service bus queue.
Anyway, I'm pretty much at a loss of how to proceed. Any ideas?
UPDATE
I've also noticed that AKS updates were rolled out the same day I started seeing performance issues.
https://github.com/Azure/AKS/blob/master/CHANGELOG.md
I wonder if this is just down to continerd differences: https://docs.microsoft.com/en-us/azure/aks/cluster-configuration#containerd-limitationsdifferences