scrapy - Terminating crawl if following an infinite website

user_2096048 0
Viewed 441

Let's imagine I have a webpage like this.

counter.php 

if(isset($_GET['count'])){
    $count = intval($_GET['count']);
    $previous = $count - 1;
    $next = $count + 1;
    ?>
    <a href="?count=<?php echo $previous;?>">< Previous</a>

    Current: <?php echo $count;?>

    <a href="?count=<?php echo $next;?>">Next ></a>
    <?
}

?>

This is an "infinite" website because you can just keep clicking next to go to the next page (the counter will just increase) or previous etc.

However, if I wanted to crawl this page and follow the links using scrapy like this, scrapy will never stop crawling.

Example spider:

urls = []  
class TestSpider(CrawlSpider):
        name = 'test'
        allowed_domains = ['example.com']
        start_urls = ['http://example.com/counter?count=1']


        rules = (
            Rule(LinkExtractor(), callback='parse_item', follow=True),
            )

        def parse_item(self, response):
            urls.append(response.url)

What kind of mechanism can I use to determine if indeed I am stuck in an infinite website and need to break out of it?

scrapy-spider scrapy web-scraping python
2 Answers

You can always break out if the page does not have ITEMS on that page, or do not have NEXT PAGE button, that means pagination has ended

class TestSpider(CrawlSpider):
        name = 'test'
        allowed_domains = ['example.com']

        def start_requests(self):
            page = 1
            yield Request("http://example.com/counter?page=%s" % (page), meta={"page": page}, callback=self.parse_item)

        def parse_item(self, response):

            #METHOD 1: check if items availble on this page         
            items = response.css("li.items")

            if items:
                #Now go to next page
                page = int(response.meta['page']) + 1
                yield Request("http://example.com/counter?page=%s" % (page), meta={"page": page}, callback=self.parse_item)
            else:
                logging.info("%s was last page" % response.url)

            #METHOD 2: check if this page has NEXT PAGE button, most websites has that          
            nextPage = response.css("a.nextpage")

            if nextPage:
                #Now go to next page
                page = int(response.meta['page']) + 1
                yield Request("http://example.com/counter?page=%s" % (page), meta={"page": page}, callback=self.parse_item)
            else:
                logging.info("%s was last page" % response.url)
user_40942310

You don't have to use Rule in the scrapy. You can first parse the page by page and then iterates all items in each page. Or you can collect all item links in the each page. For example:

urls = []
class TestSpider(CrawlSpider):
    name = 'test'
    allowed_domains = ['example.com']
    start_urls = ['http://example.com/counter?count=1']

    def parse(self, response):
        links = response.xpath('//a[@class="item"]/@href').extract()
        for link in links:
            yield Request(link, self.parse_item)
            # you can insert the item 's url here, so you dont have to yield to parse_item
            # urls.append(link)

        url, pg = response.url.split("=")# you can break infinite loop here
        if int(pg) <= 10: #We loop by page #10
            yield Request(url + "=" + str(int(pg) + 1), self.parse)

    def parse_item(self, response):
        urls.append(response.url)
user_105711820
Related