Python multiple-process forwarding requests with streaming data

Viewed 18

In test.py, I have one task to do the requests which is sending large files. As defined in offical doc, I used the streaming upload like below...

test.py

def upload():
    with open(file_name,"rb") as f:
        req = requests.post(url,data=f,headers=headers)

And here I used multiprocess.Pool to run several requests sending it to local server.

pool = Pool(3)
pool.apply_async(upload,arg=())

Then, the file in local server would need to do another request to serveral remote servers (and this is why I did multiple processing)

from fastapi import Request   
import httpx
async def uploadfile(request:Request):
    async with httpx.AsyncClient() as client:
        r = client.post(url,data=request.stream())

And in remote client, I read the stream data,and write into somewhere. The question is, as I transfer request.stream() I have to use async function in local server, which seems to stuck the work. Any other good ways to forward this stream data? Or can I form this stream data to a file in the local server and do the further job?

0 Answers
Related