I have a link to a folder which has enormous number of files that I want to download. I started downloading it single file at a time, however it's taking a very long time. Is there a way to spawn some multi-threaded processes to download maybe a batch of files simultaneously. Probably like process1 downloads the first 20 files in the folder, process2 downloads the next 20 simultaneously and so on.
Right now, I'm doing as follows:
import urllib, os
os.chdir('/directory/to/save/the/file/to')
url = 'http://urltosite/folderthathasfiles
urllib.urlretrieve(url)