Scrapy spider stops after running 3 times

SeanCedric 0
Viewed 33

I have the following Python3 code I use to scrape a website:

import scrapy

class MicrophonesSpider(scrapy.Spider):
    name = 'microphones'
    allowed_domains = ['www.marktplaats.nl']
    start_urls = ['https://www.marktplaats.nl/l/muziek-en-instrumenten/microfoons']

    def parse(self, response):
        for ad in response.xpath("//ul[@class='mp-Listings mp-Listings--list-view']/li/a"):
            yield {
                'title' : ad.xpath("..//figure/div/img/@alt").get(),
                'link' : ad.xpath("..//@href").get()
            }

        next_page = response.xpath("//nav[@class='mp-PaginationControls-pagination']//a[./span[contains(@class,'mp-svg-arrow-right--inverse')]]/@href").extract_first()

        if next_page:
            next_page = response.urljoin(next_page)
            print('\n\n' + next_page) # this line is for debugging only
            yield response.follow(url=next_page, callback=self.parse)

This code works fine, but it somehow only scrapes the first three webpages and stops after that, I'm trying to figure out why that is the case.

Becuase the response.xpath("//nav[@class='mp-PaginationControls-pagination']//a[./span[contains(@class,'mp-svg-arrow-right--inverse')]]/@href").get() line still has a valid match inside the HTML when I inspect the https://www.marktplaats.nl/l/muziek-en-instrumenten/microfoons/p/3/ webpage when I search for it inside the page source.

It also doens't gives an error it just says 2022-09-21 22:53:10 [scrapy.core.engine] INFO: Closing spider (finished) so it seems that it's not a bug.

Please note that I run the spider as scrapy crawl microphones within the Scrapy framework.

Thanks in advance.

xpath scrapy python-3.x
1 Answers

It means that what Scrapy is received is different than what you see in the browser after page 3.

One easy way to debug this is to use open_in_browser as follows:

from scrapy.utils.response import open_in_browser
...

if next_page:
    # your pagination code
else:
    open_in_browser(response) #for debug

What I see is that there is nothing after the third page. what scrapy gets

This site knows that it is bot traffic. You can verify this by removing the pagination logic and adding open_in_browser for the first page.

Moreover, if you look at the page source, that also is different.

My answer explains the WHY part of the question. As to the HOW part, I will leave it up to you.

userJkf40
Related