How to processes remaining payload when the handler get timeout in AWS lambda?

user_12977280 0
Viewed 793

I am trying to transform data from CSV to JSON in AWS lambda (using Python 3). The size of file is 65 MB, so its getting timeout before completing the process and the entire execution get fails.

I would need to know how I can handle such a case where AWS Lambda should able to process a maximum set of data within the time out period and the remaining payload should keep into an S3 bucket.

Below is the transformation code

import json
import boto3
import csv
import os
json_content = {}


def lambda_handler(event, context):
    s3_source = boto3.resource('s3')
    if event:
        fileObj=event['Records'][0]
        fileName=str(fileObj['s3']['object']['key'])
        eventTime =fileObj['eventTime']    
        fileObject= s3_source.Object('inputs3', fileName)
        data = fileObject.get()['Body'].read().decode('utf-8-sig').split()
        arr=[]
        csvreader= csv.DictReader(data)
        newFile=getFile_extensionName(fileName,extension_type)
        for row in csvreader:
            arr.append(dict(row))
        json_content['Employees']=arr
        print("Json Content is",json_content)
        s3_source.Object('s3-output', "output.json").put(Body=(bytes(json.dumps(json_content).encode('utf-8-sig'))))
        print("File Uploaded")

    return {
        'statusCode': 200,
        'fileObject':eventTime,
     }

AWS Lambda function configuration:

Memory: 640 MB

Timeout: 15 min

aws-serverless python-3.6 python-3.7 aws-lambda python-3.x
2 Answers

Since your function is timing-out, you only have two options:

  1. Increase the amount of assigned memory. This will also increase the amount of CPU assigned to the function, so it should run faster. However, this might not be enough to avoid the timeout.

or

  1. Don't use AWS Lambda.

The most common use-case for AWS Lambda functions is for small microservices, sometimes only running for a few seconds or even a fraction of a second.

If your use-case runs for over 15 minutes, then it probably isn't a good candidate for AWS Lambda.

You can look at alternatives such as running your code on an Amazon EC2 instance or using a Fargate container.

user_1747770

It looks like your function is running out of memory:

Memory Size: 1792 MB Max Memory Used: 1792

Also, it only ran for 12 minutes:

Duration: 723205.42 ms

(723 seconds ≈ 12 minutes)

Therefore, you should either:

  • Increase memory (but this costs more), or
  • Change your program so that, instead of accumulating the JSON string in memory, you continually write it out to a local disk file to /tmp/ and then upload the resulting file to Amazon S3

However, the maximum disk storage space provided to an AWS Lambda function is 512MB and it appears that your output file is bigger than this. Therefore, increasing memory would be the only option. The increased expense related to assigning more resources to the Lambda function suggests that you might be better-off using EC2 or Fargate rather than Lambda.

user_1747770
Related