I am trying to download this file of 25+ GB. The problem is that I don't have enough memory to fit it into RAM and while downloading it with chrome, I get connection errors.
I used the cli command as kaggle datasets download -d awsaf49/coco-2017-dataset but it gave me error as:
OSError: [Errno 27] File too large
I tried working with:
import requests
from tqdm import tqdm
URL = "https://storage.googleapis.com/kaggle-data-sets/857191/1462296/bundle/archive.zip?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=gcp-kaggle-com%40kaggle-161607.iam.gserviceaccount.com%2F20220502%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20220502T123414Z&X-Goog-Expires=259199&X-Goog-SignedHeaders=host&X-Goog-Signature=74f649e1bd337c9586602f42d1bc60f97f6bc1f5fc8fc76041d747adb520aadca4b8e3282cfdb16cc49164ef7ce513c5fb6c95921069385435c0e16b31247d90bb7d4d3cd7e88a2de6fe1656368f22f7ab34dbd250e3580af7f5b36968140c3a43b6f9e90da42a6cbe5921ccadf1cd7b5667d66ead4dbf92ad79d38f266250940a9449ff62a6850bfc81543f9c9a67ad6c2b1559a690eb812c192e73ed0b27d47ba97010e91a23b8dae20d2e32d68266cb943a3abfcf2bed75b37e60873aa5f6de0212e05a3e056c637c8270782876fdbb13ebd673978079d363f78bcdced64761b560df495071cfe2008d8b25fe937e35608d1263b3bca4aa53527b51a41309"
def get_size(url):
response = requests.head(url)
size = int(response.headers['Content-Length'])
return size
def download_file(url):
local_filename = "./archive.zip"
total_size = get_size(url) # total size in bytes
chunk_size = 100000000 # size of 1 chunk to download 100000000 = 100 MB
with requests.get(url, stream=True, allow_redirects=True) as r:
r.raise_for_status()
with open(local_filename, 'wb') as f:
for chunk in tqdm(r.iter_content(chunk_size = chunk_size), total = total_size // chunk_size): # Download a 100 mb chunk
#if chunk:
f.write(chunk)
return local_filename
download_file(URL)
After 4.3 GB of file, gives me the same error:
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
/tmp/ipykernel_7289/2178583065.py in <module>
24
25
---> 26 download_file(URL)
/tmp/ipykernel_7289/2178583065.py in download_file(url)
20 for chunk in tqdm(r.iter_content(chunk_size = chunk_size), total = total_size // chunk_size): # Download a 100 mb chunk
21 #if chunk:
---> 22 f.write(chunk)
23 return local_filename
24
OSError: [Errno 27] File too large
What is the solution to that? I'm using Ubuntu 20 with Python 3.7