I have a Django application with a React frontend, and need to upload multi-gigabyte files (around 8-12gb) selected from the React on a remote machine.
Currently I'm using a simple form and uploading the files via fetch with formData in the body and then reading them out of request.FILES, however I'm not very happy with this solution, since it takes over a minute to upload the files even from the same computer to itself (and I'm not entirely sure why).
Is there any way to speed up this process? The production environment for this is a gigabit local network without any external internet access so no cloud storage please.
The data seems to be fairly compressible, easily reducing it's size by 30%+ when zipped/tared.
Is there any way to compress during file uploads, or any parameters I can change on either end to speed up the process?
It doesn't seem like an 8 GB file should take over a minute to upload from the same machine to itself, and in fact should theoretically be faster than that over a gigabit network. How can I streamline this?
It'd be nice if I could make a progress bar on frontend but fetch didn't currently allow that, but if another method does that helps with the file uploads, happens to have a way to monitoring progress of the file upload, please mention it.
React 15.4, Django 1.10, Python 3.5 lots of memory, many CPU cores, and even for the server, a GPU (in case someone has some bizarre idea to use CUDA for decompression or something though I've never heard of such a thing) are available, in case there's some sort of parallelization that could help saturate the connection.