I'm trying to catch all possible errors and save page as "unhealthy". The problem is that no all errors are being sent to parse or errback. Some of them are just lost.
I've set dont_filter=True when yielding request and errback=self.error
def error(self, failure):
# log all failures
meta = failure.request.meta
website = meta['website']
if failure.check(HttpError):
# these exceptions come from HttpError spider middleware
# you can get the non-200 response
response = failure.value.response
website.set_response_code(response.status, save=False)
elif failure.check(DNSLookupError):
website.set_response_code(WebSite.RESPONSE_CODE__DNS_LOOKUP_ERROR, save=False)
elif failure.check(TimeoutError, TCPTimedOutError):
website.set_response_code(WebSite.RESPONSE_CODE__TIMEOUT, save=False)
else:
website.set_response_code(WebSite.RESPONSE_CODE__UNKNOWN, save=False)
Do you know what to do?
EDIT
Even some timeout errors like:
2020-09-07 11:27:09 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.example.com/>
Traceback (most recent call last):
File "/home/ubuntu/.virtualenvs/newswd/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/home/ubuntu/.virtualenvs/newswd/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/home/ubuntu/.virtualenvs/newswd/lib/python3.6/site-packages/scrapy/core/downloader/middleware.py", line 44, in process_request
return (yield download_func(request=request, spider=spider))
File "/home/ubuntu/.virtualenvs/newswd/lib/python3.6/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/home/ubuntu/.virtualenvs/newswd/lib/python3.6/site-packages/scrapy/core/downloader/handlers/http11.py", line 374, in _cb_timeout
raise TimeoutError("Getting %s took longer than %s seconds." % (url, timeout))
twisted.internet.error.TimeoutError: User timeout caused connection failure: Getting https://www.example.com/ took longer than 30.0 seconds..