Unable to get rid of some error raised by process_exception

user_7180194 0
Viewed 461

I'm trying not to show/get some error thrown by scrapy within process_response in RetryMiddleware. The error the script encounters when max retry limit is crossed. I used proxies within middleware. The weird thing is that the exception the script throws is already within the EXCEPTIONS_TO_RETRY list. It is completely okay that the script may sometimes cross the number of max retries without any success. However, I just do not wish to see that error even when it is there, meaning suppress or bypass it.

The error is like:

Traceback (most recent call last):
  File "middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
twisted.internet.error.TCPTimedOutError: TCP connection timed out: 10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond..

This is how process_response within RetryMiddleware looks like:

class RetryMiddleware(object):
    cus_retry = 3
    EXCEPTIONS_TO_RETRY = (defer.TimeoutError, TimeoutError, DNSLookupError, \
        ConnectionRefusedError, ConnectionDone, ConnectError, \
        ConnectionLost, TCPTimedOutError, TunnelError, ResponseFailed)

    def process_exception(self, request, exception, spider):
        if isinstance(exception, self.EXCEPTIONS_TO_RETRY) \
                and not request.meta.get('dont_retry', False):
            return self._retry(request, exception, spider)

    def _retry(self, request, reason, spider):
        retries = request.meta.get('cus_retry',0) + 1
        if retries<=self.cus_retry:
            r = request.copy()
            r.meta['cus_retry'] = retries
            r.meta['proxy'] = f'https://{ip:port}'
            r.dont_filter = True
            return r
        else:
            print("done retrying")

How can I get rid of the errors in EXCEPTIONS_TO_RETRY?

PS: The error the script encounters when max retry limit is reached no matter whatever site I choose.

scrapy middleware web-scraping python-3.x python
3 Answers

Maybe the problem is not on your side but there might be something wrong with the third party site. Maybe there is a connection error on their server or maybe it is secure so like no one can access it.

Cause the error even says that the error is with the party able it is shut down or not working properly maybe first check if the third party site is working when requested. Try contacting them if you can.

Cause the error is not in your end it's on the party's end as the error says.

This question is similar to Scrapy - Set TCP Connect Timeout

Admin1

When max retry is reached, method like parse_error() should handle any error if it is there within your spider:

def start_requests(self):
    for start_url in self.start_urls:
        yield scrapy.Request(start_url,errback=self.parse_error,callback=self.parse,dont_filter=True)

def parse_error(self, failure):
    # print(repr(failure))
    pass

However, I thought of suggesting a completely different approach here. If you go the following route, you don't need any custom middleware at all. Everything including retrying logic is already there within the spider.

class mySpider(scrapy.Spider):
    name = "myspider"
    start_urls = [
        "some url",
    ]

    proxies = [] #list of proxies here
    max_retries = 5
    retry_urls = {}

    def parse_error(self, failure):
        proxy = f'https://{ip:port}'
        retry_url = failure.request.url
        if retry_url not in self.retry_urls:
            self.retry_urls[retry_url] = 1
        else:
            self.retry_urls[retry_url] += 1
        
        if self.retry_urls[retry_url] <= self.max_retries:
            yield scrapy.Request(retry_url,callback=self.parse,meta={"proxy":proxy,"download_timeout":10}, errback=self.parse_error,dont_filter=True)
        else:
            print("gave up retrying")

    def start_requests(self):
        for start_url in self.start_urls:
            proxy = f'https://{ip:port}'
            yield scrapy.Request(start_url,callback=self.parse,meta={"proxy":proxy,"download_timeout":10},errback=self.parse_error,dont_filter=True)

    def parse(self,response):
        for item in response.css().getall():
            print(item)

Don't forget to add the following line to get the aforesaid result from the above suggestion:

custom_settings = {
    'DOWNLOADER_MIDDLEWARES': {
        'scrapy.downloadermiddlewares.retry.RetryMiddleware': None,
    }
}

I'm using scrapy 2.3.0 by the way.

user_105685310

Try fixing the code in the scraper itself. Sometimes have a bad parse function can lead to a error of the kind you're describing. Once I fixed the code, it went away for me.

user_140842350
Related