Scrapy errback doesn't catch all errors

Viewed 116

I'm trying to catch all possible errors and save page as "unhealthy". The problem is that no all errors are being sent to parse or errback. Some of them are just lost.

I've set dont_filter=True when yielding request and errback=self.error

def error(self, failure):

    # log all failures
    meta = failure.request.meta
    website = meta['website']

    if failure.check(HttpError):
        # these exceptions come from HttpError spider middleware
        # you can get the non-200 response
        response = failure.value.response
        website.set_response_code(response.status, save=False)

    elif failure.check(DNSLookupError):
        website.set_response_code(WebSite.RESPONSE_CODE__DNS_LOOKUP_ERROR, save=False)

    elif failure.check(TimeoutError, TCPTimedOutError):
        website.set_response_code(WebSite.RESPONSE_CODE__TIMEOUT, save=False)
    else:
        website.set_response_code(WebSite.RESPONSE_CODE__UNKNOWN, save=False)

Do you know what to do?

EDIT

Even some timeout errors like:

2020-09-07 11:27:09 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.example.com/>
Traceback (most recent call last):
  File "/home/ubuntu/.virtualenvs/newswd/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/ubuntu/.virtualenvs/newswd/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/ubuntu/.virtualenvs/newswd/lib/python3.6/site-packages/scrapy/core/downloader/middleware.py", line 44, in process_request
    return (yield download_func(request=request, spider=spider))
  File "/home/ubuntu/.virtualenvs/newswd/lib/python3.6/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/ubuntu/.virtualenvs/newswd/lib/python3.6/site-packages/scrapy/core/downloader/handlers/http11.py", line 374, in _cb_timeout
    raise TimeoutError("Getting %s took longer than %s seconds." % (url, timeout))
twisted.internet.error.TimeoutError: User timeout caused connection failure: Getting https://www.example.com/ took longer than 30.0 seconds..
0 Answers
Related