Scrapy freeze on connection timeout

user_475763 0
Viewed 604

I wrote a scrapy crawler that uses an Internet connection that is pretty unreliable. This is something that is a given. I cannot easily or cheaply change it - once in a while the Internet connection will be lost and after a few seconds or so it will be restored.

I observe behaviour where a Scrapy 18.4 crawler would freeze indefinitely without printing any error messages. It stops reacting to Ctrl+C, which makes me think this happens somewhere pretty deep in the reactor stack, though I cannot be sure.

There are absolutely no error messages which makes things rather hopeless to debug.

Question: Would anyone have any clues as to how to debug this problem? I don't really have any meaningful logs to attach for the reasons laid out above.

scrapy twisted web-crawler timeout python
1 Answers

You can set up a class for timeout and then run your code in a try except. Something like this:

import signal

class timeout:
    def __init__(self, seconds=1, error_message='Timeout'):
        self.seconds = seconds
        self.error_message = error_message
    def handle_timeout(self, signum, frame):
        raise TimeoutError(self.error_message)
    def __enter__(self):
        signal.signal(signal.SIGALRM, self.handle_timeout)
        signal.alarm(self.seconds)
    def __exit__(self, type, value, traceback):
        signal.alarm(0)

def run_crawl():
    while True:
        print("This runs")

try:
    with timeout(seconds=3):
        run_crawl()
except Exception as e:
    print(e)

Note: Since this uses signal it will only work on UNIX-based systems.

enter image description here

If you want it to go back to running the crawler (auto-restart), then you can just put it all in an infinite loop.

while True:
    print("Restarting spider")
    try:
        with timeout(seconds=3):
            run_crawl()
    except Exception as e:
        print(e)

This is all assuming that you can just keep restarting the bot after x seconds without major negative results. IE if you are constantly scraping the same page(s) over and over again, then this would work pretty seamlessly.

However, if you are scraping a very long list [once] and just want it to finish without an error, then it would work less well, but could still be used by setting x to a number that represents an amount of time greater than the duration of the entire process when it executes successfully (this is true no matter the length of the execution of the program - don't set x to 3 seconds if you are scraping one site that takes 7 seconds to complete, and the same is true if you are doing 5 that take 30 seconds or 500 that take 5 minutes, you will need to set x to an amount greater than that duration.

The reason that I specifically separated your bot having a quick completion time vs. not is that if it fails during a loop with a timeout of 30 seconds then you lose, on average, 15 seconds if it fails, but if you have a 30 minute execution time then you would lose, on average, 15 minutes when it fails, and if your internet is going out every 15 minutes, on average, then you would have it failing a vast majority of the time and you'll need to look more into debugging the problem and actually solving it instead of working around it, as this "solution" should definitely be considered a workaround.

user_32964220
Related