Best practices of uploading a file to S3 and metadata to RDS?

user_9153516 0
Viewed 598

Context

I'm building a mock service to learn AWS. I want a user to be able to upload a sound file (which other users can listen to). To do this I need the sound file to be uploaded to S3 and metadata such as file name, name of uploader, length, S3 ID to RDS. It is preferable that the user uploads directly to S3 with a signed URL instead of doubling the data transfered by first uploading it to my server and from there to S3.

Optimally this would be transactional but from what I have gathered there's no functionality for that given. In order to implement this and minimize the risk of the cases where the file being successfully uploaded to S3 but not the metadata to RDS and vice versa my best guess is as follows:

My solution

With words: First is an attempt to upload the file to S3 with a key (uuid) I generate locally or server-side. If this is successful I make a request to my API to upload the metadata including the key to RDS. If this is unsuccessful I remove the object from S3.

With code:

uuid = get_uuid_from_server(); 
s3Client.putObject({.., key: uuid, ..}, function(err, data) {
   if (err) {
    reject(err);
   } else {
    resolve(data);
    // Upload metadata to RDS through API-call to EC2 server. Remove s3 object with key: 
       uuid if this call is unsuccessful
   }
  });

As I'm learning, my approaches are seldom the best practices but I was unable to find any good information on this particular problem. Is my approach/solution above in line with best practices?

Bonus question: is it beneficial for security purposes to generate the file's key (uuid) server-side instead of client-side?

amazon-rds amazon-web-services amazon-s3
1 Answers

Here are 2 approaches that you can pick, assuming the client is a web browser or mobile app.

1. Use your server as a proxy to S3.

Your server acts as a proxy between your clients and S3, you have full control of the upload flow, control the supported file types and can inspect file contents, for example: to make sure the file is a correct sound file, before uploading to S3.

2. Use your server to create pre-signed upload URLs

In this approach, your client first requests server to create a single or multiple (for multi-part upload) pre-signed URLs. Clients then upload to your S3 using those URLs. Your server can save those URLs to keep track later.

To be notified when the upload finishes successfully or unsuccessfully, you can either

(1) Ask clients to call another API,e.g: /ack after the upload finishes for a particular signed URL. If this API is not called after some time, e.g: 1 hour, you can check with S3 and delete the file accordingly. You can do this because you have the signed URL stored in your DB at the start of the upload.

or

(2) Make use of S3 events. You can configure ObjectCreated event in S3, which is fired whenever an object is created, and send all the events to a queue in SQS, and have your server process each event from there. This way, you do not rely on clients to update your server after an upload finishes. S3 will notify your server accordingly, for all successful uploads.

user_18020130
Related