How to show progress on aiohttp POST with both form data and file

user_16611796 0
Viewed 432

Ok this is my working code:

data = aiohttp.FormData()
data.add_field('title', title)
data.add_field('author', user)
data.add_field('upload_file', open(path, 'rb'))

up_session = aiohttp.ClientSession()
async with up_session.post(url="http://example.com/upload.php", data=data) as response:
    resp = await response.text()

resp = json.loads(resp)

What I want to know is how to add some sort progress monitoring to it. I can't find any sort of callback in the docs nor a generator that works with MultipartWriter (FormData is just a helper for MultipartWriter). I'm losing my mind here. Thanks in advance.

EDIT: I used to get it with request and requeststoolbelt (MultipartEncoder, MultipartEncoderMonitor) but those are not async, and aiohttp is such a complete library i can't believe you cant do that.

    encoder = MultipartEncoder(
    fields={
    'upload_file': (ntpath.basename(path), open(path, 'rb'), 
    'application/octet-stream'),
    'title': str(''),
    'author': str(user),
    })
    upload_data = MultipartEncoderMonitor(encoder, upload_progress)

    headers={'Content-Type': upload_data.content_type}
    headers.update(http_headers)
    
    r_2 = session.post(url=url_domain + "/repository/repository_ajax.php?action=upload", data=upload_data)

def upload_progress(monitor):
    print (str(monitor.len) + " - " + "{:.2f}%".format(monitor.bytes_read/monitor.len))
aiohttp post asynchronous upload python
1 Answers

I did find a solution, a little hacky but it works and it's kind of general for other libraries that don't provide good progress tracking but accepts binary streams. It's a wrapper around BufferedReader. Technically counts the bytes as they're read instead of send, but for a progress bar it's kind of the same.

from pathlib import Path
from io import BufferedReader
from aiohttp import ClientSession


class ProgressFileReader(BufferedReader):
    def __init__(self, filename, read_callback=None):
        f = open(filename, "rb")
        self.__read_callback = read_callback
        super().__init__(raw=f)
        self.length = Path(filename).stat().st_size

    def read(self, size=None):
        calc_sz = size
        if not calc_sz:
            calc_sz = self.length - self.tell()
        if self.__read_callback:
            self.__read_callback(self.tell(), self.length)
        return super(ProgressFileReader, self).read(size)


def progress_callback(current, total):
    print(100 * current // total)


async def main():
    with ProgressFileReader(filename="./file.jpg", read_callback=progress_callback) as file:
        upload_payload = {
            "foo": "bar",
            "file": file,
        }
        async with ClientSession() as session:
            async with session.post("http://example.org", data=upload_payload) as response:
                resp = await response.text()

The wrapper itself is sync, but it's no biggie and it can be made async if you wrap around aiofiles instead of the builtin "open". Credits for the wrapper class idea to some other stackoverflow question that I can't find now.

user_166117960
Related