I have a question, what is the best practice between these two approaches considering we need to upload large files?
First Approach:
Reading a file & creating chunks (slice of a file i.e. blob) on UI and sending each chunk to the backend from the UI.
On the Backend read each chunk as streams & send each chunk to the desired place (s3).
If we receive all chunks of data in the backend, assemble the chunks at the desired place (s3).
Benefits:
- Fewer chances of timeouts (considering large file).
- Ease of retries which reduces chances of errors (we can resend the chunk in case of failure).
Disadvantage:
- It will consume more memory on the UI (creating chunks).
Second Approach:
Reading the whole file from the UI, reading the file chunk by chunk on the backend & uploading the chunks at the desired place (s3), and assembling the parts.
Advantages:
- Less memory on the UI.
Disadvantages:
- More timeouts & more error-prone.
Could you please help me in deciding the best approach? I understand the solution depends upon the use case to use case, but please note we want to enable users to upload larger files.