Why my Scrapy Response.follow is not following the links and returns empty?

Plinio Leal 0
Viewed 29

Below you can see my spider (in this case is just a test). As i test, i first run the parse_remedios as the main parse an it returns the results. But when i use as the second function (parse_remedios, like in the code below) the results are empty. I know that has something about the response.follow is not running well. Any ideas?

import scrapy

class RemediosSpider(scrapy.Spider):
name = 'remedios'
allowed_domains = ['www.drogariaspachecopacheco.com.br']
start_urls = ['https://www.drogariaspacheco.com.br/clorana%2025mg']


def parse(self, response):
    print(response.url)
    for link in response.css('.collection-link::attr(href)'):
        yield response.follow(link.get(), callback=self.parse_remedios)

def parse_remedios(self, response):
    resultado = response.css('.container-fluid')
    yield {
    'nome' : resultado.css('.productName::text').get(),
    'preco' : resultado.css('.skuBestPrice::text').get() ,
    'link' : response.url,
    'sku' : resultado.css('.skuReference::text').get()
    }
scrapy python
1 Answers

The problem is with your allowed_domains. Scrapy is filtering all of the links after the start_urls because they do not match any of the domains in your allowed_domains list. I have corrected in the example below

you can see it in the output logs

2022-09-15 21:38:24 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to 'www.drogariaspacheco.com.br': <GET https://www.drogariaspacheco.com.br/clorana-25mg-sanofi-aventis-30-comprimidos/p>

import scrapy

class RemediosSpider(scrapy.Spider):
    name = 'remedios'
    allowed_domains = ['drogariaspacheco.com.br']
    start_urls = ['https://www.drogariaspacheco.com.br/clorana%2025mg']


    def parse(self, response):
        for link in response.css('.collection-link::attr(href)'):
            yield response.follow(link.get(), callback=self.parse_remedios)

    def parse_remedios(self, response):
        resultado = response.css('.container-fluid')
        yield {
        'nome' : resultado.css('.productName::text').get(),
        'preco' : resultado.css('.skuBestPrice::text').get() ,
        'link' : response.url,
        'sku' : resultado.css('.skuReference::text').get()
        }
userphOc10
Related