How to wait for all s3 files to be uploaded

asdasf 0
Viewed 33

I have a Step Function State Machine that executes some lambdas in a flow.

I only want to run the State Machine after I upload the 10 required files for my flow, which is the tricky part.

  1. 10 files are manually uploaded in S3 bucket
  2. S3 upload triggers a notification to EventBridge (need to wait for all new CSVs to be uploaded)
  3. EventBridge starts the State Machine

I think my current flow will not work because it will call the state machine 10 times for each file upload...

I know how to use S3 file upload to trigger a state machine like in this example but I don't see how to make sure that ALL the files are uploaded before triggering the state machine ONCE.

Is this possible to achieve? Any ideas?

amazon-web-services
1 Answers

I think my current flow will not work because it will call the state machine 10 times for each file upload

That's right. Everytime an object is uploaded, the S3 event will trigger EventBridge which will invoke the step function.

To achieve your desired workflow, swap out the EventBridge rule with a lambda function. Have the lambda function check the total size of the bucket, and if the condition is met (10 files have been uploaded) invoke the Step Function directly.

userN1ee0
Related