How can I download medias from the web most quickly? stream, or asynchronous requests?

Viewed 31

I have many urls now, there is a media file behind every url. Then, how can I download them most quickly? As far as I know, The way is most slow way:

import requests

for url,path in zip(urls,paths):
    with open(path,'wb') as f:
        f.write(requests.get(url).content)

And, If I use the way, It will be more quick:

import requests

for url,path in zip(urls,paths):
    with requests.get(url,stream=True) as r, open(path,'wb') as f:
        for chunk in r.iter_content(chunk_size=1024):
            f.write(chunk)

The way do the same thing, make it more quick:

from requests_futures.sessions import FuturesSession
from concurrent.futures import as_completed

session = FutureSession()

futures = []
for url,path in zip(urls,paths):
    future = session.get(url)
    future.path = path
    futures.append(future)

for future in as_completed(futures):
    r = future.result()
    with open(future.path,'wb') as f:
        f.write(r.content)

So, what I want to know is, which one is more quick? And, Can I combine them?

0 Answers
Related