Load Balancer Monitoring Surge Queue Length

user_7663643 0
Viewed 2487

Can anyone explain me what's happening with my AWS Load Balancer?

I see the metric Surge Queue Length with two lines growing "together" in a cumulative way: Surge Queue Length

From the documentation, it says that this is the queue of requests on the Load Balancer to be processed by the backend (EC2 instance) and all the trouble-shooting advices that I found points to a performance issue on the backend, but in my case the instance is healthy (CPU, memory, disk i/o, etc.. everything is fine).

This Load Balancer belongs to a Elastic Beanstalk worker environment with only one instance. And every time I deploy a new version, it seems that the Surge Queue Length is purged.

Can anyone explain me why this cumulative queue is growing even if my backend instance is fine? And why this is purged when I deploy?

amazon-elastic-beanstalk amazon-elb amazon-web-services
1 Answers

Even if the EC2 instance in the back end is healthy looking (CPU, memory, disk, etc...), it could be behind in processing the requests sent by the ELB. This can happen if (like in my case) the EC2 is running under an Elastic Beanstalk environment with Docker, where the EC2 instance can only run one Docker container. In this case, the Docker container running the app can't process all of the incoming requests, but since it's inside an isolated environment (the container), it can't use all of the available resources within the EC2 instance.

In my case, I had to scale up my EC2 instances inside the Autoscaling Group (sitting behind the ELB) even when my EC2 instances report that they are using 5% CPU. After scaling up (CPU Utilization went down to 1%), my performance issues went away.

Hope this helps

user_25550460
Related