handling large web uploads for photo and videos

Question

My use case involves uploading thousands of full quality photo and video files using browser to S3 and Wasabi storage accounts. Currently we are compression it on client's browser and right now we did it using dropzonejs which handles uploading. Right now its compressed before being uploaded to server.

However, that's what we need to change. We need to upload original quality photos and that's where it gets stuck as we cannot upload files more than 3-4 Gbs using Dropzonejs. Not sure what prevents it but we are struggling to find solution for this. We face problem randomly with memory limit in Chrome which crashes and need to restart process again. With original quality photo we assume this would not work as we will be talking for at least 10 to 15 gbs of data at least.

What would you recommend for this kind of use case where we need to upload video & photos in original quality sometimes a single photo could be taking as much as 40Mbs+. And video several Gbs.

How does Google photos manages this? We need something like that.

user_787171 · Answer

Chunking...

someone already have demo https://github.com/dropzone/dropzone/blob/main/test/test-sites/2-integrations/aws-s3-multipart.html

but, i think 4GB is the max file size that Chrome will accept(And I think chrome has the highest limit compare to other browsers). which means you need to use other method to upload such as ftp, stream, scp etc... or ask your clients to slice the files themselves before uploading through their browser.

Or create a custom executable that bundles with S3 client, and let your clients use that

user_18399373 · Answer

dropzonejs supports chunking and paralleling did you use them? do you compress files by dropzone like this: https://stackoverflow.com/a/51699311/18399373

user_4535739 · Answer

Do not compress on client side. It actually increases memory usage on the browser session. To my experience uploading original image from the browser uses least amount of memory as the browser should only read enough from the file to send the data, as long as you're not loading the picture locally to show the thumbnails.

I was able to upload GBs of images to S3 with client side compression turned off. I was able to upload a single 20GB video file to S3, upload 200 videos totaling over 13GB using S3 Chunk upload. Chunk upload should increase, not decrease browser memory usage and was implemented to hand transmission failures for large files.

user_14356967 · Answer

You can have a huge performance improvement by writing your client to upload directly to S3 by providing a signed url(s) and have them skip the server as the middle man:

S3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html

GCP: https://cloud.google.com/blog/products/storage-data-transfer/uploading-images-directly-to-cloud-storage-by-using-signed-url

user_17321778 · Answer

I would recommend using presigned url from S3. In our Project we generate presigned url by giving bucket name, path name, bucket access to upload and expiry time. So now the user can upload the file easily directly to S3. AWS will take care of all the networking issues, only condition is you should have a good internet.

handling large web uploads for photo and videos

5 Answers