I have a Node.js (v 12.x) Express site running as a Lambda function behind a single instance of a load balancer. The load balancer is set to us-east-1a and us-east-1b. Everything is in the same region. The reason I'm using a load balancer is because that is the only way you can route a Route 53 domain to a Lambda function.
When I first fired everything up, there was a minimum 6 second, up to 10 second delay before receiving any content at all from the web site. Even if I did a simple page refresh, the time it took to get any content back from my server was at least 6 seconds, even if the browser cached it. However all the files were eventually delivered fine with zero errors in the Chrome DevTools window, it just took on average about 10 seconds before you could see the page.
To speed things up I upped the memory for the Lambda function from 128 MB to 1024 MB, and set the memory size for the single Express app within it to that same value. I then provisioned 3 concurrency units and 5 reserved concurrency units. I also upped the timeout value to the maximum, 29 seconds. This fixed the speed problem and dropped the response time below 3 seconds. However, I immediately started getting a 502 error in the browser for every file delivered except for the first few requested. The browser does retry quickly and then gets most of the files on the second pass. Unfortunately it never gets all of them and there are always few of them that are never downloaded and show up in the Chrome DevTools window as aborted 502 errors (ERR_ABORTED),leaving the page improperly rendered since one or more CSS files were never successfully retrieved.
- If I look in the Lambda monitor screen, none of the graphs show errors.
- If I look in the Cloudwatch logs, I don't see any errors there either.
- However, if I download the load balancer access logs from the S3 bucket that holds them, I see a ton of "Lambda Throttling" errors like these, after the first 2 or 3 successfully serviced requests:
h2 2021-11-18T02:56:38.095418Z app/MAIN-BALANCER/506da40318eeac61a552 {REMOVED}:18034 - 0.004 0.009 0.000 502 - 44 594 "GET https://neobandbattles.com:443/javascripts/google-closure/base.js HTTP/2.0" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:us-east-1:977323236257:targetgroup/EXPRESS-LAMBDA-FUNCTION/2dfd7eac9069e4feb81 "Root={REMOVED}" "neobandbattles.com" "arn:{REMOVED}" 0 2021-11-18T02:56:38.082000Z "forward" "-" "LambdaThrottling" "-" "-" "-" "-"
This web site gets at most 2 or 3 visitors per day. I've read the documents on Lambda throttling and I can't see any way I'm hitting the 1000 concurrent requests limit or anything close to it. Can someone tell me how to fix this? Note, there only about 22 files total requested from the web site, that are necessary to render the page. It's not like rendering the page requires a huge number of requests that might trigger a throttling error. Something else is seriously wrong here.
Here are the relevant parts of my serverless.yml file, since it sets the Lambda function configuration:
service: my-web-site
projectDir: ./
provider:
name: aws
runtime: nodejs12.x
timeout: 29 # Max is 29 seconds.
tracing:
lambda: true
iamRoleStatements:
- Effect: "Allow"
Action:
- dynamodb:Query
- dynamodb:Scan
- dynamodb:GetItem
- dynamodb:PutItem
- dynamodb:UpdateItem
- dynamodb:DeleteItem
Resource: "{REMOVED}"
stage: dev
region: us-east-1
memorySize: 1024
environment: {REMOVED}
functions:
app:
description: My web site
handler: app/app.handler
events:
- http:
path: /
method: ANY
cors: true
- http:
path: /{proxy+}
method: ANY
cors: true
memorySize: 1024
provisionedConcurrency: 3
reservedConcurrency: 5
tracing: PassThrough
package:
exclude:
- .git/**