How to identify what is causing a throttling exception in AWS lambda and how to handle the same

user_226906 0
Viewed 1603

I have a node.js lambda that subscribes to a SQS queue and makes a REST API call to a microservice for every message from SQS.

Now as the lambda is executing I get the following exception

{
  "message": "Rate exceeded",
  "code": "ThrottlingException",
  "time": "2020-03-27T23:11:42.209Z",
  "requestId": "69f5c441-409h-493s-8da5-2361c5055fd8",
  "statusCode": 400,
  "retryable": true
}

I am unable to understand what AWS service is throwing this exception. Is this exception thrown by my lambda or some other AWS service ?

Also what can be done to avoid this ?

aws-lambda amazon-web-services throttling
2 Answers

As per AWS, it says:

Throttling is intended to protect your resources and downstream applications. Though Lambda automatically scales to accommodate your incoming traffic, your function can still be throttled for various reasons.

Technically it could be a programming error or it's possible that throttles that you're seeing aren't on your Lambda function. Throttles can also occur on API calls during your function's invocation. I will put some points from AWS documentation on how to identify what is causing the throttling:

  1. Verify if you see throttling messages in Amazon CloudWatch Logs but no corresponding data points in the Lambda Throttles metrics. If there are no Lambda Throttles metrics, the throttling is happening on API calls in your Lambda function code.
  2. Check your function code for any throttled API calls. If certain API calls are throttled, be sure to use exponential backoff in your code to retry the API calls.
  3. If you determine that you need a higher transactions per second (TPS) quota for an API call, you can request a service quota increase, if the quota is adjustable.

Source

user_47045100

I solved this by simply pausing the function execution for a few milliseconds after each AWS API call. Here's an example (notice await sleep(..) calls:

for (; ;) {
  const describeLogStreamsResponse = await cloudWatchLogs.describeLogStreams(describeLogStreamsRequest).promise();
  await sleep(700);
  for (let i = 0; i < describeLogStreamsResponse.logStreams.length; i++) {
    const logStream = describeLogStreamsResponse.logStreams[i];
   ...
      try {
        await cloudWatchLogs.deleteLogStream(deleteLogStreamRequest).promise();
        await sleep(1000);
      } 
       ....
const sleep = (milliseconds) => {
  return new Promise(resolve => setTimeout(resolve, milliseconds));
};
user_28019090
Related