AWS Load balancer causing slow API requests

user_1558119 0
Viewed 1702

My API requests are extremely slow when called through https over my AWS elastic load balancer. What's the best way to figure out exactly what I need to do to make my load balancer more efficient? I'm using the load balancer specifically for HTTPs, and running an ec2 instance with an HTTP server behind it.

AWS elastic load balancer in eu-central-1a, eu-central-1b.
EC2 instance in eu-central-1a is a c5.xlarge which hosts an http server.

When I make API requests from the East Coast US to the load balancer via an HTTPS call, it works but it's horribly slow. Best case scenario it's 400ms and it spikes up to 800-1500ms about 20% of the time.

When I make API requests from the East Coast US directly to the EC2 instance via HTTP totally bypassing the load balancer it's super quick. Almost always 130-140ms and rarely spikes.

NOTE: I'm using the load balancer for the HTTPS support only, I don't really care (right now) about actual load balancing across multiple backends. For now, I just have one backend. All of my incoming calls will be coming from the East Coast US and my server must stay in Europe for the time being. So yes, I could move my server to the East Coast US so that calls are saying in one region, but I do not want to do that for various other reasons.

Questions and possible solutions:

  • I chose an Application Load Balancer, is that right?
  • How can I tell if my configuration is to blame for it being slow?
  • Should I move my load balancer to East Coast US where the clients will be calling from? And leave the EC2 instance in eu-central-1a?
  • My ec2 instance (c5.xlarge) has had absolutely nothing running in it except the http server so I know it's not getting overloaded with CPU and memory usage. Is my instance type bad for being connected to a load balancer?

Problem solved:

  • Got some help from a Dev ops guys. In Route 53 console, he removed the two A records that I created in the setup wizard and replaced them with a CNAME record direct to my alias. What was odd, the wizard wouldn't let me do that. I had to use the "old version" of Route 53 where it let me type directly into the record.
  • Performance is great now. The first request when it's establishing the secure connection takes a few hundred milliseconds, but every request after that is ~130ms. And that's from East Coast US to Frankfurt Germany. Working as expected now.
  • I tried moving my load balancer to another region, but AWS doesn't allow that.
amazon-elb amazon-web-services amazon-ec2 https
1 Answers

For the lowest latency you will want the shortest distance between the client and the servers, using cloudping I can see even directly between AWS regions on the AWS backbone (this is AWS private network) there is a latency of 95.22ms. Over public internet this will be even more.

If you are unable to currently move the instances to another region, the performance of latency will still apply. However, I would suggest taking a look at CloudFront which is the AWS CDN service.

The Amazon CloudFront content delivery network (CDN) is massively scaled and globally distributed. The CloudFront network has 216 points of presence (PoPs), and leverages the highly-resilient Amazon backbone network for superior performance and availability for your end users.

A major feature of CloudFront is providing local caching of front end objects (images, css, static HTML), if you can cache any of these objects your users experience will significantly improve. Many sites will use this service to cache many static or rarely changing parts of their site such as the homepage.

If parts of static page are dynamic, can you look at loading them asynchronously after initial page load? This gives the user the perception the site is loading whilst parts of the page are still loading afterwards.

You will also gain the benefit of performance improvements made to CloudFront such as its utilization of the AWS backbone to gain improved latency between the users PoP location and the target endpoint.

A couple of links that might be interesting for you:

user_134609330
Related