# Get the content type of a URL
def get_url_type(url: str) -> str:
r = urlopen(url)
header = r.headers
return header.get_content_type()
Would the following only fetch the headers or would it fetch the whole document?
I'm using this to check if it is or not an html page, to avoid downloading (big) files.
It appears that only the headers and a bit of the body is retrieved in the urlopen() call, and the rest is gotten upon the read().
In testing this with something like wireshark, you can see that even if you call urllib.urlopen('VeryLargeFile') you still only receive headers and a few packets of body initially.
However, if you only wish to know the size of a page or its content type you can use requests:
import requests
rh = requests.head(url)
header = rh.headers
content_type = header.get('content-type')
print(content_type)
# text/html; charset=UTF-8
I've check out urllib source code, it's lazily evaluated.
Firstly it will only read header. And it will only read body when you call read in the response object
import requests
url = 'http://www.google.com'
%timeit -n 25 requests.head(url).headers
%timeit -n 25 requests.request('HEAD', url).headers
%timeit -n 25 requests.get(url).headers
Output:
185 ms ± 13.4 ms per loop (mean ± std. dev. of 7 runs, 25 loops each)
171 ms ± 18.6 ms per loop (mean ± std. dev. of 7 runs, 25 loops each)
242 ms ± 29.7 ms per loop (mean ± std. dev. of 7 runs, 25 loops each
Making a specific 'HEAD' type request seems to be fastest.