Scrapy: Predefined parse method in Pycharm

Seraph 0
Viewed 31

I learning Scrapy, and using Pycharm verison 2021.3.1 Community Edition. When creating the parse() method for the Spider class, Pycharm suggests defining the parse() method with two parameters (response) and (kwargs):

    def parse(self, response, **kwargs):
        pass

However, according to the the Scrapy documentation, the parse() method is defined with one parameter (response).

    def parse(self, response):
        pass

Questions:

  1. Is there anything wrong with Pycharm's predefined parse() method? Are there any cases where the parse() method would pass two arguments?

  2. If there is something wrong with this predefined parse() method, how can the suggestion snippet be edited?

Thanks for any help!

pycharm scrapy python
1 Answers

As you mentioned that you are a beginner, def parse(self, response): would work. You can ignore PyCharm's warning or use def parse(self, response, **kwargs):

Once you learn more, you will feel the need of passing data from one callback function to another. This functionality can be achieved using cb_kwargs.

For example, see this code:

def parse(self, response):
    current_page = response.css('.footer::text').get()
    product_link = response.css('.prod::attr(href)').get()

    yield scrapy.Request(product_link, callback=self.parse_product,
        cb_kwargs={'page':current_page}


def parse_product(self, response, page): #this callback receives additional param
    self.logger.log(f'Current page :{page})

I hope this makes sense.

You can see the documentation for more details.

userJkf40
Related