How to deploy projects with larger dependencies and exceeding deployment limits on AWS Lambda function?

Viewed 1000

I have to deploy a python project on AWS Lambda function. When I create its zip package it occupies a memory of around 80 MB (Lambda allows upto 50 MB). Also I cannot upload it to s3 because the memory size of the uncompressed package is around 284 MB (S3 allows upto 250 MB). Any idea how to tackle this problem or Is there any alternative for it?

2 Answers

The ZIP file is rather not an issue, as you would split your dependencies into lambda layers. Each layer can have 50 MB in zip, so you could have 2 layers.

The issue is that total unpacked size of your function + layers still must be less then 250 MB. There is no golden solution for this. You have to loose 34 MB from your 284 MB. You can go over content of your dependencies and start manually removing any documentation files, text files or anything that is non-essential. If the dependencies involve compiled shared objects you can use tools such as strip to reduce their size.

Basically, its a try and see approach what can be removed from your dependencies and what not. This can be time-consuming and troublesome process. Or you may re-factor your application to get rid some of dependencies as well.

Is there any alternative for it?

The alternative is not to use lambda function. The closest alternative is probably an ECS Fargate service.

Unfortunately, you can't upload a deployment to Lambda that has an uncompressed size of 250MB even when you use Layers (Read Note). There's a workaround to this, but it will impact your Lambda performance a lot.

  • Write a driver function in Lambda for your code.
  • This driver function will download your code from S3 into ephemeral Lambda Storage (/tmp/) and then run it.

Eventual solution is to use ECS Fargate (Serverless) or AWS Batch.

Adding a code sample for Lambda based solution:

import boto3
import os 
import subprocess
import sys

sys.path.insert(1, '/tmp')

def download_code_from_s3(bucketName, file_path):
    s3_resource = boto3.resource('s3')
    bucket = s3_resource.Bucket(bucketName) 
    bucket.download_file(file_path, os.path.join('/tmp/', file_path))

def lambda_handler(event, context):
    #Download zip file from S3 to /tmp/
    download_code_from_s3('bucket_name', 'code.zip')
    subprocess.run(['unzip', '/tmp/code.zip', '-d', '/tmp/'])
    
    #import and run the actual handler, while code.py is the entry point of the project
    from code import my_handler
    return my_handler(event, context)

Above code is deployed to Lambda function. Its responsibility is to download the zip file, unzip it, import the entry point and then run it. Unzipped size of contents must not exceed 512 MB.

Related