scrapy.Request not going through

Question

The crawling process seems to ignore and/or not execute the line yield scrapy.Request(property_file, callback=self.parse_property). The first scrapy.Request in def start_requests goes through and executed properly, but not one in def parse_navpage as seen here.

import scrapy

class SmartproxySpider(scrapy.Spider):
    name = "scrape_zoopla"
    allowed_domains = ['zoopla.co.uk']

    def start_requests(self):
        # Read source from file
        navpage_file = f"file:///C:/Users/user/PycharmProjects/ScrapeZoopla/ScrapeZoopla/ScrapeZoopla/spiders/html_source/navpage/NavPage_1.html"
        yield scrapy.Request(navpage_file, callback=self.parse_navpage)

    def parse_navpage(self, response):
        listings = response.xpath("//div[starts-with(@data-testid, 'search-result_listing_')]")

        for listing in listings:
            listing_url = listing.xpath(
                "//a[@data-testid='listing-details-link']/@href").getall()  # List of property urls
            break
        print(listing_url) #Works

        property_file = f"file:///C:/Users/user/PycharmProjects/ScrapeZoopla/ScrapeZoopla/ScrapeZoopla/spiders/html_source/properties/Property_1.html"
        print("BEFORE YIELD")
        yield scrapy.Request(property_file, callback=self.parse_property) #Not going through
        print("AFTER YIELD")

    def parse_property(self, response):
        print("PARSE PROPERTY")
        print(response.url)
        print("PARSE PROPERTY AFTER URL")

Running scrapy crawl scrape_zoopla in the command returns:

2022-09-10 20:38:24 [scrapy.core.engine] DEBUG: Crawled (200)  (referer: None)
BEFORE YIELD
AFTER YIELD
2022-09-10 20:38:24 [scrapy.core.engine] INFO: Closing spider (finished)

Both scrapy.Requests requested local files and only the first one worked. The files exist and properly display the pages and in case one of them does not the crawler would return error "No such file or directory" and likely be interrupted. It seems here the crawler just passed right through the request, not even gone through it, and returned no error. What is the error here?

userphOc1 · Answer

This is a total shot in the dark but you could try sending both requests from your start_requests method. I honestly don't see why this would work but It might be worth a shot.

import scrapy

class SmartproxySpider(scrapy.Spider):
    name = "scraoe_zoopla"
    allowed_domains = ['zoopla.co.uk']

    def start_requests(self):
        # Read source from file
        navpage_file = f"file:///C:/Users/user/PycharmProjects/ScrapeZoopla/ScrapeZoopla/ScrapeZoopla/spiders/html_source/navpage/NavPage_1.html"
        property_file = f"file:///C:/Users/user/PycharmProjects/ScrapeZoopla/ScrapeZoopla/ScrapeZoopla/spiders/html_source/properties/Property_1.html"
        yield scrapy.Request(navpage_file, callback=self.parse_navpage)
        yield scrapy.Request(property_file, callback=self.parse_property)

    def parse_navpage(self, response):
        listings = response.xpath("//div[starts-with(@data-testid, 'search-result_listing_')]")

        for listing in listings:
            listing_url = listing.xpath(
                "//a[@data-testid='listing-details-link']/@href").getall()  # List of property urls
            break
        print(listing_url) #Works


    def parse_property(self, response):
        print("PARSE PROPERTY")
        print(response.url)
        print("PARSE PROPERTY AFTER URL")

Update

It just dawned on me why this is happening. It is because you have the allowed_domains attribute set but the request you are making is on your local file system which naturally is not going to match the allowed domain.

Scrapy assumes that all of the initial urls sent from start_requests are permitted and therefore doesn't do any verification for those, but all subsequent parse methods check against the allowed_domains attribute.

Just remove that line from the top of your spider class and your original structure should work fine.

scrapy.Request not going through

1 Answers

Update