Alternate solutions to Lambda Triggers for file processing?

user_18116488 0
Viewed 74

We have a use case where some process drops a file at a S3 location and we want to pick it and process it as soon as it lands. Lambda solves this perfectly but our infra team decided it no longer supports Lambda’s, literally days before we were about to release. So now I m trying to find an alternate solution. We do have databricks so we can run a job every two mins but that means having a cluster running all the time which we’d like to avoid.

Our infra team has decided they are not going to support any AWS service other than S3, so can’t use anything under AWS framework.

Any ideas on how I can do that?

Thanks!

aws-lambda triggers amazon-s3
1 Answers

S3 can push to lambda, event bridge, sns or sqs only. All of these are AWS services. So if you can use only S3, that's out of question already w/o even starting to think about destination.

if it must be out of aws, then it has to be out of aws. Something sends bytes to S3. That something "knows" if it was a success and where it was sent. So it can trigger your services hosted anywhere via some api call.

Back in a day before cloud you would host some HTTP api endpoint to send the trigger or if you have some kind of private network, could send local HTTP or SSH or any other protocol based message to place where your processing code runs.

You need to send event about object landed on s3 somehow.

user_17371580
Related