Scrapy callback problem: to scrape the follow-up page of an item I need this item's (auto_increment) ID in the database

Wolf D. 0
Viewed 39

I'm trying to do the following:

  1. scrape some items (which include URLs) from some page.
  2. save each item to a mysql database
  3. get the (auto_increment) ID of the previously added record
  4. scrape the page pointed to by the URL above (#1)
  5. save all items on this page with a reference (foreign key) to the ID defined in #3

That looks to me like a very common problem which should have a reasonably simple solution.

The "item_scraped" signal looked like the obvious solution since it provides a callback for when an item has been fully processed.

However, it appears that it's impossible to start another Request from within the callback.

I read here that some people suggested using Spider Middlewares but I don't see how, especially since Spider Middlewares are not supposed to start Requests themselves.

I'm using Scrapy 2.6.2 on Python 3.8.

Thanks for your help.

My simplified code below, without error processing etc.

The spider

import scrapy
from scrapy import signals

class MySpider(scrapy.Spider):
    name = 'MySpider'
    allowed_domains = ['www.somedomain.com']
    start_urls = [
        'https://www.somedomain.com/someurl/', 
        ]

    @classmethod
    def from_crawler(cls, crawler, *args, **kwargs):
        spider = cls()
        # requests the callback once an item is scraped
        crawler.signals.connect(spider.item_scraped, signals.item_scraped)
        return spider

    def parse(self, response):
        for node in response.xpath(".//tr/td[@class='foo']"):
            yield {
                'a': ...
                'b': ...
                'url': ...
            }

    def item_scraped(self, item, response, spider):
###
### UNFORTUNATELY THIS IS NOT POSSIBLE
###
        # forwards the new ID as a meta variable
        yield Request(item['url'], meta={'id':item['id']}, callback=self.parse_page)

    def parse_page(self, response):
        for node in response.xpath("//div[@class='soemthing']//a"):
            yield {
                'id': response.meta['id'],
                'title': ...
            }

And the pipeline:

import mysql.connector

class WinebotPipeline(object):

    def __init__(self):
        self.conf = {'host': ..., 'user': ..., 'password': ..., 'database': ...}
        self.connection = self.mysql_connect()
        self.cursor = self.connection.cursor()
    
    def mysql_connect(self):
        return mysql.connector.connect(**self.conf)

    def process_item(self, item, spider):
        sql = "insert into `some_table` (`a`, `b`, `url`) values (%s, %s, %s)" 
        self.cursor.execute(sql, list(item.values()))
        self.connection.commit()

        # id of last inserted item
        item['id'] = self.cursor.lastrowid
        return item
scrapy web-scraping python
1 Answers

An easier way of doing this would be to get all of the info from the first page then use the url to yield a new request with your parse_page method as the callback and the scraped data as the callback kwargs. Then once you have scraped the necessary info from the second url you can add it to the results from the first page and yield the completed item. Then you can do all of your database entries all at once.

Spider

import scrapy
from scrapy import signals

class MySpider(scrapy.Spider):
    name = 'MySpider'
    allowed_domains = ['www.somedomain.com']
    start_urls = [
        'https://www.somedomain.com/someurl/', 
        ]

    def parse(self, response):
        for node in response.xpath(".//tr/td[@class='foo']"):
            cb_kwargs = {'a': ..., 'b': ...,   'url': ...}
            yield scrapy.Request(cb_kwargs['url'], 
                                 callback=self.parse_page, 
                                 cb_kwargs=cb_kwargs)
            

    def parse_page(self, response, **kwargs):
        for node in response.xpath("//div[@class='soemthing']//a"):
            kwargs.update({'title': ...})
            yield kwargs

Pipeline

import mysql.connector

class WinebotPipeline(object):

    def __init__(self):
        self.conf = {'host': ..., 'user': ..., 'password': ..., 'database': ...}
        self.connection = self.mysql_connect()
        self.cursor = self.connection.cursor()
    
    def mysql_connect(self):
        return mysql.connector.connect(**self.conf)

    def process_item(self, item, spider):
        sql = "insert into `some_table` (`a`, `b`, `url`, 'title') values (%s, %s, %s, %s)" 
        self.cursor.execute(sql, list(item.values()))
        self.connection.commit()
        return item
userphOc10
Related