Can I merge value get from webpage by scrapy into on item?

Question

I'm building spider by scrapy. I defined item like below:

class MyItem(scrapy.Item):
    # define the fields for your item here like:
    title = scrapy.Field()
    usage=scrapy.Field()
    storage=scrapy.Field()

and the title comes from the first page, usage comes from the second page and the storage comes from the third page

No I try to fetch value from webpage like below

def parse_item(self, response):
    links=response.xpath('somepath')
    for link in links:
        name=link.xpath('text()').get()
        href= 'mydomain'+link.xpath('@href').get()
        yield scrapy.Request(href,callback=self.parse_info,meta={"title":name})

def parse_info(self,response):
    item=MyItem()
    item["title"]=response.request.meta['title']

    # usage
    item['usage']=response.xpath("sompath").get().strip()

    # storage
    storagelink = response.xpath("somepath/@href").get()
    item['usage'] = scrapy.Request(storagelink ,callback=self.parse_storage,meta={"title":name})
    print(item)
    return item

def parse_storage(self,response):
    return response.xpath("somepath/text()").get().strip()

but I can't get the usage value, the console always output:

'usage': ,

Why? And how can I merge there data into items? Thanks!

SuperUser · Answer

You set item['usage'] as the request, even if it was right scrapy is asynchronous so it wouldn't work right.

Declare the item in the for loop, pass it through the methods and yield it only in the last one. Also you can use cb_kwargs), as it's more convenient.

def parse_item(self, response):
    links = response.xpath('somepath')
    for link in links:
        item = MyItem()
        item['title'] = link.xpath('text()').get()
        href = 'mydomain' + link.xpath('@href').get()
        yield scrapy.Request(href, callback=self.parse_info, cb_kwargs={'item': item})

def parse_info(self, response, item):
    # usage
    item['usage'] = response.xpath("sompath").get().strip()

    # storage
    storagelink = response.xpath("somepath/@href").get()
    scrapy.Request(storagelink, callback=self.parse_storage, cb_kwargs={'item': item})

def parse_storage(self, response, item):
    item['storage'] = response.xpath("somepath/text()").get().strip()
    print(item)
    yield item

Can I merge value get from webpage by scrapy into on item?

1 Answers