AWS Lambda - Body Size is Too Large Error, but Body Size is Under Limit

user_1553183 0
Viewed 9420

I am using a lambda function to service a REST API. In one endpoint I am getting "body size is too long" printed to the cloudwatch log.

The response I get from the function is status code 502 with response body { "message": "Internal server error" }. If I call the same endpoint but use a filter, the response body size is 2.26MB and works. This rules out that I am hitting asynchronous response body limit.

The response body size when it errors out is 5622338 bytes (5.36 MB).

This is how I am calculating the response size (python 2.7):

import urllib2
...

out = {}
resp = urllib2.urlopen(req)
out.statusCode = resp.getcode()
out.body = resp.read()
print("num bytes: " + str(len(bytearray(out.body, 'utf-8'))))

The advertised max response body size is for synchronous invocations is 6MB. From what I understand, I should not be receiving the error. https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html

Other information:

Duration: 22809.11 ms Billed Duration: 22810 ms Memory Size: 138 MB Max Memory Used: 129 MB Init Duration: 1322.71 ms

Any help would be appreciated.

Update 4/22/21

After further research I found that the lambda function errors out if the size of the response is 1,048,574 bytes (0.999998 MB) or more.

If the response is 1,048,573 bytes (0.999997 MB) or less it works.

This is how I am returning responses. I hard code the view function to return a bytearray of a specific size. Ex.

return bytearray(1048573)

I turned on logging for the AWS Gateway Stage that I am using, and the following error is getting written to the log. It implies that the function is erroring out. Not the invocation of the function: Lambda execution failed with status 200 due to customer function error: body size is too long.

It's my understanding that the AWS Lambda Functions have a max response size of 6MB and the AWS Gateways have a max response size of 10MB.

AWS Lambda: Invocation payload response https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html

API Gateway: API Gateway quotas for configuring and running REST API -> Payload size https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html

Am I misunderstanding the limits?

aws-lambda python-2.7 amazon-web-services rest api
2 Answers

I created a new lambda function for python 3.7 and found that it returns a more descriptive error. I was able to determine that the lambda function adds around 1 MB to the size of the response after it leaves the lambda function which explains why it was erroring out in the lambda function, but not in the endpoint code. In one case it added .82MB and another it add .98MB. It seems to be based off the size of the response. I suspect the response is getting base64 encoded, URL encoded, or something similar. I did not find documentation that could answer this and the responses were not encoded in any way on the receiving end.

Error Message returned by lambda function build with python 3.7:

Response payload size (8198440 bytes) exceeded maximum allowed payload size (6291556 bytes).

Lambda Function Code:

import requests, base64, json

def lambda_handler(event, context):
    
    headers = {...}
    url = ...
    
    response = requests.get(url, headers=headers)
    
    print (len(response.content))
    print (len(response.content.decode("utf-8")))

    
    return response.content.decode('UTF-8')

Below I have the size of the responses when printed inside the lambda function and the size of the response as determined by lambda in the error message.

I used these two print statements to determine the size of the response and these two print statements would always return the same length. Response.content is a byte array so my thought process is that getting the length of it will return the number of bytes in the response: print (len(response.content)) print (len(response.content.decode("utf-8")))

Ex 1.

Size when printed inside the lambda function:

7,165,488 (6.8335 MB)

Size as defined in the error message:

8,198,440 (7.8186 MB)

Extra:

1,032,952 (0.9851 MB)

Error Message:

Response payload size (8198440 bytes) exceeded maximum allowed payload size (6291556 bytes).

Ex 2. Size when printed inside the lambda function:

5,622,338 (5.3619 MB)

Size as defined in the error message:

6,482,232 (6.1819 MB)

Extra:

859,894 (0.820059 MB)

Error Message:

Response payload size (6482232 bytes) exceeded maximum allowed payload size (6291556 bytes).

I have decided to lower the soft limit in the endpoint code by another 1 MB (4.7MB) to prevent hitting the limit in the lambda function.

user_15531830

Tried the same in golang. events.APIGatewayV2HTTPResponse.Body can be ~5.300.000 bytes (didn't test the exact limit). Value was measured inside the lambda function (len(resp.Body))

user_45813740
Related