No space left on device - AWS Lambda + Sharp

user_4369379 0
Viewed 198

I am trying to convert SVG to TIFF using sharp. My NodeJs application is deployed on AWS Lambda. As the AWS Lambda /tmp directory can have maximum size of 512 MB, for the large files I am getting error "No space left on device".

I tried to search internet but could not find any good solution.

Can you please suggest, if it is possible to use sharp on application deployed on AWS Lambda to process large files? If yes the how?

Otherwise, if you can suggest any alternate way to achieve it on AWS - May be using some other services?

sharp aws-lambda node.js amazon-web-services
1 Answers

As far as I can tell you have three options:

  1. Mount a EFS volume to your Lambda
  2. Do stream processing
  3. Store the converted image in memory

Option #1

Mounting an EFS volume to your Lambda instance is effectively like adding a hard drive to your Lambda. But be aware, that the disk is shared between instances of your Lambda and persists after your Lambda is shutdown.

Documentation: https://aws.amazon.com/blogs/compute/using-amazon-efs-for-aws-lambda-in-your-serverless-applications/

Option #2

This is the more complicated option and might not work with all formats. But instead of writing every bit to disk, you can process the image in chunks and then write those chunks to S3 for example. This way your Lambda does not need to store any data on disk at all and does not need a lot of memory because you are processing the image in small chunks.

Option #3

Instead of writing the image to disk (toFile()) you could write the result into a buffer (toBuffer()). Then you use this buffer to write the result to S3 for example. This way no bits touch the disk and your only constrain is the maximum Lambda memory of 10.240 MB.

Recommendation: Before you try any solution, make sure that you are deleting your converted images. The /tmp directory persists between requests for as long as your Lambda instance is alive. That means that the same Lambda instance could be invoked 20 times and write a new file to /tmp every time. If you do not clean up those files after processing them , your "No disk space left" error might be due to old files that not have been deleted.

user_1908230
Related