Move files from S3 to AWS EFS on the fly

user_1701168 0
Viewed 5415

I am building an application where the user can upload images, for this I am using S3 as files storage.

In other area of the application there is some process deployed on EC2 that need to use the uploaded images. This process need the images multiple times (it generate some report with it) and it's in part of multiple EC2 - using elastic beanstalk.

The process doesn't need all the images at once, but need some subset of it every job it gets (depend the parameters it gets).

Every ec2 instance is doing an independent job - they are not sharing file between them but they might need the same uploaded images.

What I am doing now is to download all the images from s3 to the EC2 machine because it's need the files locally. I have read that EFS can be mounted to an EC2 and then I can access it like it was a local storage.

I did not found any example of uploading directly to EFS with nodejs (or other lang) but I found a way to transfer file from S3 to EFS - "DataSync". https://docs.aws.amazon.com/efs/latest/ug/transfer-data-to-efs.html

So I have 3 questions about it:

  1. It is true that I can't upload directly to EFS from my application? (nodesjs + express)
  2. After I move files to EFS, will I able to use it exactly like it in the local storage of the ec2?
  3. Is it a good idea to move file from s3 to efs all the time or there is other solution to the problem I described?
amazon-efs node.js amazon-web-services storage amazon-s3
2 Answers

For this exact situation, we use https://github.com/kahing/goofys

It's very reliable, and additionally, offers the ability to mount S3 buckets as folders on any device - windows, mac as well as of course Linux.

Works outside of the AWS cloud 'boundary' too - great for developer laptops.

Downside is that it does /not/ work in a Lambda context, but you can't have everything!

user_76560910

Trigger a Lambda to call an ECS task when the file is uploaded to s3. The ECS task starts and mounts the EFS volume and copies the file from s3 to the EFS. This wont run into problems with really large files with Lambda getting timed out. I dont have the code but would be interested if someone already has this solution coded.

user_47389500
Related