Python requests.post very slow

user_6225449 0
Viewed 123

I am currently using requests to push a file via API to my SaaS product.

    file = {'file': open(file_name1, 'rb')}
headers = {'Authorization': 'Bearer ' + app_key}
response = requests.post(url=push_chunk_url ,files=file, headers=headers)

However, the upload speed is very slow (~1mbs) despite 150mbs upload available to the browser.

This response from 6 years ago suggests that the block size of httplib may be responsible: https://stackoverflow.com/a/39518613

Since that was written Python 3.7 introduced a blocksize parameter to httplib replacement http.client https://docs.python.org/3/library/http.client.html

I can't work out though how to possible pass the blocksize parameter from requests to set this higher and hopefully solve this issue.

python-requests python
1 Answers

According to the Github Issue(Requests is slow to upload large files) it's not possible to passed down the blocksize parameter down to the http.client.HTTPConnection. The workaround solution is to monkey patch the HTTPConnection.__init__.__defaults__ with your block size(here we are using 64*1024).

from http.client import HTTPConnection
HTTPConnection.__init__.__defaults__ = tuple(
    x if x != 8192 else 64*1024
    for x in HTTPConnection.__init__.__defaults__
)
user_66994470
Related