I have many urls now, there is a media file behind every url. Then, how can I download them most quickly? As far as I know, The way is most slow way:
import requests
for url,path in zip(urls,paths):
with open(path,'wb') as f:
f.write(requests.get(url).content)
And, If I use the way, It will be more quick:
import requests
for url,path in zip(urls,paths):
with requests.get(url,stream=True) as r, open(path,'wb') as f:
for chunk in r.iter_content(chunk_size=1024):
f.write(chunk)
The way do the same thing, make it more quick:
from requests_futures.sessions import FuturesSession
from concurrent.futures import as_completed
session = FutureSession()
futures = []
for url,path in zip(urls,paths):
future = session.get(url)
future.path = path
futures.append(future)
for future in as_completed(futures):
r = future.result()
with open(future.path,'wb') as f:
f.write(r.content)
So, what I want to know is, which one is more quick? And, Can I combine them?