I am attempting to create a simple Flask endpoint for uploading files via POST or PUT. I want the filename in the URL, and then to (after the request headers) just stream the raw file data in the request.
I also need to be able to upload files slightly larger than 2GB, and I need to do this without storing the entire file in memory. At first, this seemed simple enough:
@application.route("/upload/<filename>", methods=['POST', 'PUT'])
def upload(filename):
# Authorization and sanity checks skipped.
filename = secure_filename(filename)
fileFullPath = os.path.join(application.config['UPLOAD_FOLDER'], filename)
with open(fileFullPath, 'wb') as f:
copyfileobj(request.stream, f)
return jsonify({'filename': filename})
With a multipart/formdata upload, I can simply call .save() on the file.
However, any file I upload seems to have a different checksum (well, sha256sum, on the server then on the source). When uploading a standard text file, newlines seem to be getting stripped. Binary files seem to be getting mangled in other strange ways.
I am sending Content-Type: application/octet-stream when uploading to try to make Flask treat all uploads as binary. Is request.stream (a proxy to wsgi.input) opened as non-binary? I can't seem to figure that out from the Flask code. How can I stream the request data, in raw binary format, to a file on disk?
I'm open to hacks; this is for a test project (so I'm also not interested in hearing how sending this as formdata would be better, or how this isn't a good way to upload files, etc.)
I am testing this via:
curl -H 'Content-Type: application/octet-stream' -H 'Authorization: ...' -X PUT --data @/path/to/test/file.name https://test.example.com/upload/file.name