I'd like to be able to detect when all parts of an S3 multipart upload have been uploaded.
Context
I'm working on a Backend application that sits between a Frontend and an S3 bucket.
When the Frontend needs to upload a large file it makes a call to the Backend (step 1). The latter initiates a multipart upload in S3, generates a bunch of presinged URLs, and hands them to the Frontend (steps 2 - 5). The Frontend uploads segment data directly to S3 (steps 6, 10).
S3 multipart uploads need to be explicitly completed. One obvious way to perform it would be to make another call from the Frontend to the Backend to notify about the fact that all parts have been uploaded. But if possible I'd like to avoid that extra call.
A possible solution: S3 Event Notifications
I have S3 Event Notifications enabled on the S3 bucket so whenever something happens, it notifies an SNS topic which in turn calls the Backend.
If the bucket sent S3 notifications after each part is done uploading, I could use those in the Backend to see if it's time to complete the upload (steps 7 - 9, 11 - 14).
But although some folks claim (one, two) that it's the case, I wasn't able to reproduce it.
For proof of concept, I used this guide from Amazon to upload a file using aws s3api create-multipart-upload, several aws s3api upload-part, and aws s3api complete-multipart-upload. I would expect to get a notification after each upload-part, but I only got a single "s3:ObjectCreated:CompleteMultipartUpload" after, well, complete-multipart-upload.
My bucket is configured to send notification for all object creation events: "s3:ObjectCreated:*".
Questions
- Is it possible to somehow instruct S3 to send notifications upon upload of each part?
- Are there any other mechanisms to find out in the Backend that all parts have been uploaded?
- Maybe what I want is complete nonsense and even if there was a way to implement it, it would bring significant drawbacks?