Whenever I change parse function scrapy doesn't work and throws error?

user_6910995 0
Viewed 49
from scrapy import Spider
from selenium import webdriver
from scrapy.selector import Selector
from scrapy.http import Request

from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options


class BookSeleniumSpider(Spider):
    name = 'book_selenium'
    allowed_domains = ['books.toscrape.com']



    def start_req(self):
        s = Service('C:\\Users\\aps\\Documents\\chromedriver.exe')
        chrome_options = webdriver.ChromeOptions()
        chrome_options.add_argument('--no-sandbox')
        self.driver = webdriver.Chrome(service=s, chrome_options=chrome_options)

        # Get the site we want to start scraping
        self.driver.get('http://books.toscrape.com')


        sel = Selector(text=self.driver.page_source)
        books = sel.xpath('//h3/a/@href').extract()


        for book in books:
            url = 'http://books.toscrape.com/' + book
            print(url)

    def parse_book(self, response):
        pass

As I change the parse function to start_req, it stops working. But When I changes it back to parse it works fine. I don't know why. Can anyone explain that to me

scrapy web-scraping selenium web-crawler python
1 Answers

When scrapy requests are created they are intialized with a callback function for processing the results. Unless the user explicitly identifies this callback function then the default function used is the parse method. So when you change the name of the method, it will throw an error because the callback no longer exists.

If for some reason you did want to change the name, one way to do so that requires very little effort would be to assign the parse attribute to the function name inside of the class. For Example:

class BookSeleniumSpider(Spider):
    name = 'book_selenium'
    allowed_domains = ['books.toscrape.com']

    def start_req(self):
        s = Service('C:\\Users\\aps\\Documents\\chromedriver.exe')
        chrome_options = webdriver.ChromeOptions()
        chrome_options.add_argument('--no-sandbox')
        self.driver = webdriver.Chrome(service=s, chrome_options=chrome_options)

        # Get the site we want to start scraping
        self.driver.get('http://books.toscrape.com')

        sel = Selector(text=self.driver.page_source)
        books = sel.xpath('//h3/a/@href').extract()
        for book in books:
            url = 'http://books.toscrape.com/' + book
            print(url)

    def parse_book(self, response):
        pass
        ...

    ...
    ...
    parse = parse_book
user_178294510
Related