How to avoid timeout from AWS API Gateway and Lambda?

user_9917132 0
Viewed 5970

I have an API endpoint on AWS API Gateway with AWS Lambda (Python & Flask) to store some data from a JSON file. e.g) curl -X POST http://www.xxx.yyy/store -d @zzz.json

However, when I tried executing the API with a bigger JSON file, I encountered a timeout error. Through my investigation, the maximum timeout setting for Lambda is 300 seconds, and API Gateway is 29 seconds. The maximum timeout for Lambda 300 sec sounds fine, but 29 seconds sounds too short. What kind of things could be a solution? The JSON data can be split by id, but it needs to be sent as one file.

EDIT: Sure I can't change the number. Any suggestion to solve this problem using another technology/system design pattern? I can't change the input, though.

EDIT2: Currently, the Lambda function has validation based on JSON scheme, parse into models, and save into database. Any suggestions?

aws-api-gateway aws-lambda flask amazon-web-services python
3 Answers

Is there anyway you can update your Lambda function to hand off to another process?

By decoupling you could for example do the following:

API Gateway -> Lambda (Perform any mandatory action, then store in S3 as a blob) -> S3 -> Another Lambda to process.

user_134609330

Uploading files with lambdas can be tricky and a direct upload is not recommended unless the file size is under the limits.

Warning currently:

  • API Gateway has a payload limit of 10 MB
  • API Gateway has Maximum timeout of 30 s
  • Lambda has an invocation payload (request and response) limit of 6 MB

The best approach is a basically a two step process:

  1. The client app makes an HTTP request to the lambda to get an upload URL. The lambda returns a pre-signed POST URL to S3
  2. The client post the file using the pre-signed URL

API Gateway limits : https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html Lambda limits: https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html

user_19106370

The timeout value cannot be increased:

Resource or operation: Integration timeout

Default quota: 50 milliseconds - 29 seconds for all integration types, including Lambda, Lambda proxy, HTTP, HTTP proxy, and AWS integrations.

Can be increased: Not for the lower or upper bounds.

Source: https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html

user_98358720
Related