How to check whether a URL is downloadable or not?

user_12635677 0
Viewed 4309

How do I check whether the given url is downloadable or not using Python?

It should return True if it is downloadable else False

An example of a non-downloadable url: www.google.com

Note: I am not speaking about contents of the URL and saving it as a web page.

What is a downloadable URL?

If you redirect to a URL and if a file starts to download, then it is a downloadable url

Example: https://drive.google.com/uc?id=1QOmVDpd8hcVYqqUXDXf68UMDWQZP0wQV&export=download

Note: It downloads the stackoverflow annual survey 2019 data set.

python-requests web-scraping python-3.x url python
4 Answers

This can be done in using the popular requests library

import requests
url = 'https://www.google.com'
headers=requests.head(url).headers
downloadable = 'attachment' in headers.get('Content-Disposition', '')

Content Disposition Header reference

user_54046940

So I tried searching for a better way, the site link which I was checking was a bit tricky most stackoverflow answers mentioned about using head request to get response header, but the site I was checking returned 404 error.When I use get request the whole file is downloaded before outputing the header.My friend suggested me a solution of using the parameter stream=True and that really got worked.

import requests 
r = requests.get(link, stream=True)
print(r.headers)
user_130339810

On HTTP protocol level itself, there is no distinction between downloadable and non-downloadable URL. There is an HTTP request and there is a subsequent response. Response body can be a binary file, HTML, image etc..

You can just request the HTTP response header and look for Content-Type: and decide whether you want to consider that content-type as downloadable or non-downloadable.

user_75636030

Downloadable files must have Content-Length in headers :

import requests
r = requests.get(url, stream=True)

try:
    print(r.headers['content-length'])
except:
    print("Not Downloadable")
user_166241440
Related