Django rest framework large file upload

user_14886355 0
Viewed 1834

Is there an easy way to upload large files from the client side to a django rest framework endpoint. In my application, users will be uploading very large files (>4gb). Browsers have a upload limit, here's the chart.

enter image description here

My current idea is to upload the file in chunks from the client side and receive the chunks from the rest endpoint. But how will I do that? I saw some libraries like - resumable.js, tus.js, flow.js etc. But how will I handle the chunks in the backend? Is there any library that is actively maintained for a problem like this? Please help me.

large-file-upload django-rest-framework django-file-upload large-files django
3 Answers

Maybe this module could help: https://github.com/jkeifer/drf-chunked-upload. The module is utilized into a sample django app at the link, with example code for implementation. Here is the typical usage case the module provides, without the sample code for simplicity (code is at the link if you want it):

  1. An initial PUT request is sent to the url linked to ChunkedUploadView (or any subclass) with the first chunk of the file. The name of the chunk file can be overriden in the view (class attribute field_name).

  2. In return, the server will respond with the url of the upload, and the current offset.

3 Repeatedly PUT subsequent chunks to the url returned from the server.

  1. Server will continue responding with the url and current offset.

  2. Finally, when upload is completed, POST a request to the returned url. This request must include the checksum (hex) of the entire file.

  3. If everything is OK, server will response with status code 200 and the data returned in the method get_response_data (if any).

  4. If you want to upload a file as a single chunk, this is also possible! Simply make the first request a POST and include the checksum digest for the file. You don't need to include the Content-Range header if uploading a whole file.

Based on these instructions, it seems that the server handles the upload by tracking offsets of the chunk through received headers ("Content-Range"), as well as its url, storing the uploaded chunks in .part files. It then responds like so:

   {
        'id': 'f64ebd67-83a3-45b6-8acd-c749ea1ed4cd'
        'url': 'https://your-host/<path_to_view>/f64ebd67-83a3-45b6-8acd-c749ea1ed4cd',
        'file': 'https://your-host/<path_to_file>/f64ebd67-83a3-45b6-8acd-c749ea1ed4cd.part',
        'filename': 'example.bin',
        'offset': 10000,
        `created_at`: '2021-05-18T17:12:50.318718Z',
        'status': 1,
        'completed_at': None,
        'user': 1
    }

When the full file is uploaded as determined by the recieved headers, the .part files are combined into the final upload. This also allows you to resume uploads if they are interuptted, because the existing .part files persist until the upload finishes.

user_81895990
user_127761160

you should do it with celery tasks.

take a look at this link. it explains how to upload a file using django and celery.

user_114878720
Related