Scrapy ignoring the content of second page

Question

I've written a tiny scraper in python scrapy to parse different names from a webpage. The page has traversed 4 more pages through pagination. The total names throughout the pages are 46 but it is scraping 36 names.

The scraper is supposed to skip the content of first landing pages but using parse_start_url argument in my scraper I've handled it.

However, the problem I'm facing at this moment with this scraper is that It surprisingly skips the content of second page and parse all the rest, I meant first page, third page, fourth page and so on. Why it is happening and how to deal with that? Thanks in advance.

Here is the script I'm trying with:

import scrapy

class DataokSpider(scrapy.Spider):

    name = "dataoksp"
    start_urls = ["https://data.ok.gov/browse?page=1&f[0]=bundle_name%3ADataset&f[1]=im_field_categories%3A4191"]

    def parse(self, response):
        for link in response.css('.pagination .pager-item a'):
            new_link = link.css("::attr(href)").extract_first()
            yield scrapy.Request(url=response.urljoin(new_link), callback=self.target_page)

    def target_page(self, response):
        parse_start_url = self.target_page  # I used this argument to capture the content of first page
        for titles in response.css('.title a'):
            name = titles.css("::text").extract_first()
            yield {'Name':name}

user_9189799 · Answer

The solution turns out to be very easy. I've fixed it already.

import scrapy

class DataokSpider(scrapy.Spider):

    name = "dataoksp"
    start_urls = ["https://data.ok.gov/browse?f[0]=bundle_name%3ADataset&f[1]=im_field_categories%3A4191"]

    def parse(self, response):
        for f_link in self.start_urls:
            yield response.follow(url=f_link, callback=self.target_page) #this is line which fixes the issue

        for link in response.css('.pagination .pager-item a'):
            new_link = link.css("::attr(href)").extract_first()
            yield response.follow(url=new_link, callback=self.target_page)

    def target_page(self, response):
        for titles in response.css('.title a'):
            name = titles.css("::text").extract_first()
            yield {'Name':name}

Now it gives me all the results.

Scrapy ignoring the content of second page

2 Answers