How to break out of crawl if certain condition encountered in Scrapy

user_2096048 0
Viewed 141

For the purposes of an application I'm working on, I need scrapy to break out of the crawl and start crawling again from a particular, arbitrary URL.

The intended behaviour is for scrapy to just back to a particular URL which can be supplied in an argument if a particular condition is satisfied.

I'm using CrawlSpider but can't figure out how to achieve this:

class MyCrawlSpider(CrawlSpider):
    name = 'mycrawlspider'
    initial_url = ""

    def __init__(self, initial_url, *args, **kwargs):
        self.initial_url = initial_url
        domain = "mydomain.com"
        self.start_urls = [initial_url]
        self.allowed_domains = [domain]
        self.rules = (
            Rule(LinkExtractor(allow=[r"^http[s]?://(www.)?" + domain + "/.*"]), callback='parse_item', follow=True),
        )

        super(MyCrawlSpider, self)._compile_rules()


    def parse_item(self, response):
        if(some_condition is True):
            # force scrapy to go back to home page and recrawl
            print("Should break out")

        else:
           print("Just carry on")

I tried to place

return scrapy.Request(self.initial_url, callback=self.parse_item)

in the branch of someCondition is True but without success. Would hugely appreciate some help, been working on trying to figure this out for hours.

scrapy web-crawler python
1 Answers

you could make a custom exception that you handle appropriately, like so...

Please feel free to edit with the appropriate syntax for CrawlSpider

class RestartException(Exception):
    pass

class MyCrawlSpider(CrawlSpider):
    name = 'mycrawlspider'
    initial_url = ""

    def __init__(self, initial_url, *args, **kwargs):
        self.initial_url = initial_url
        domain = "mydomain.com"
        self.start_urls = [initial_url]
        self.allowed_domains = [domain]
        self.rules = (
            Rule(LinkExtractor(allow=[r"^http[s]?://(www.)?" + domain + "/.*"]), callback='parse_item', follow=True),
        )

        super(MyCrawlSpider, self)._compile_rules()


    def parse_item(self, response):
        if(some_condition is True):

            print("Should break out")
            raise RestartException("We're restarting now")

        else:
           print("Just carry on")

siteName = "http://whatever.com"
crawler = MyCrawlSpider(siteName)           
while True:
    try:
        #idk how you start this thing, but do that

        crawler.run()
        break
    except RestartException as err:
        print(err.args)
        crawler.something = err.args
        continue

print("I'm done!")
user_48022590
Related