We have a simple website with links to large files. There are hundreds of thousands of links, and the file sizes range from a few MB to a few GB. The total size of all the files is several TB.
We use a Python script with the requests library to check the validity of each URL. We use "requests.get(url).status_code" and look for 404's.
I'm concerned that requests.get() downloads each entire file, which we don't need. The url's point to large files totaling several terabytes. If the script downloads each file on every request.get(), it will run very slowly and use far too many resources.
Is it possible, in requests or some other library, to get just the status code without downloading the target?
One final note: The url targets are binary files, not HTML. So tricks to limit download size of websites-such as requesting just the HTML header-probably won't work here.