Scrapy - Correct Selector for dynamically created field

user_8667315 0
Viewed 233

I'm working on a web scraper and I am having trouble with grabbing the correct selector. Here's my code:

# -*- coding: utf-8 -*-
import scrapy
import pandas as pd
from ..items import HomedepotpricespiderItem
from scrapy.http import Request


class HomedepotspiderSpider(scrapy.Spider):
    name = 'homeDepotSpider'
    allowed_domains = ['homedepot.com']


    



    start_urls = ['https://www.homedepot.com/pep/304660691']#.format(omsID = omsID)
        #for omsID in omsList]

    def parse(self, response):

    #call home depot function
        for item in self.parseHomeDepot(response):
            yield item

        pass

    def parseHomeDepot(self, response):

        #get top level item
        items = response.css('#zone-a-product')
        for product in items:
            item = HomedepotpricespiderItem()

    
    #get the price
            productPrice = product.xpath('//div[@class="price-format__main-price"]/span/text()').getall()

    #get rid of all the stuff i dont need
   
        
            item['productPrice'] = productPrice
            
            yield item

So with my current selector it looks like it is grabbing the price of these items. enter image description here

because my output is:

'productPrice': ['$',
                  '2167',
                  '49',
                  '$',
                  '1798',
                  '00',
                  '$',
                  '2698',
                  '00',
                  '$',
                  '2099',
                  '99',
                  '$',
                  '2968',
                  '00',
                  '$',
                  '2294',
                  '99',
                  '$',
                  '2068',
                  '00',
                  '$',
                  '1649',
                  '99',
                  '$',
                  '2399',
                  '00',
                  '$',
                  '1649',
                  '99',
                  '$',
                  '1549',
                  '99',
                  '$',
                  '1799',
                  '99',
                  '$',
                  '3360',
                  '89',
                  '$',
                  '2899',
                  '95',
                  '$',
                  '3699',
                  '00',
                  '$',
                  '2719',
                  '96',
                  '$',
                  '1954',
                  '99',
                  '$',
                  '2699',
                  '00',
                  '$',
                  '2294',
                  '96',
                  '$',
                  '3149',
                  '00',
                  '$',
                  '3499',
                  '00',
                  '$',
                  '3749',
                  '00',
                  '$',
                  '4999',
                  '00',
                  '$',
                  '2799',
                  '99'],

when the correct output should be: 2099

Additionally, I don't think that my selector is even grabbing the price of the item at all.

scrapy web-scraping css html python
3 Answers

First, you don't need to use getall() if you want to receive a SINGLE value (use get() instead. Another reason that your expression doesn't work because you should use relative path (for your product node):

product.xpath('.//div[@class="price-format__main-price"]/span/text()').getall()

Next there is no easy way to get a correctly formatted price with a single XPath expression (in fact you can use concat() for this) because integer and decimal parts of the price are separated by spans. The easiest way (for me) will be to get whole value and next format it to get decimal part:

product_price = response.xpath('normalize-space(//div[@id="zone-a-product"]//div[@class="price"])').re_first(r'\$(.+)')
# product_price is 209999

Update Try this XPath instead:

response.xpath('normalize-space(//div[@class="price"])').re_first(r'\$(.+)')
user_3477670

First of all class you are trying to reach is not unique for that specific price tag and second even if your class or id were unique you still need to use extract_first() or probably get() as mentioned in another answer to extract first value.

# "price-detailed__wrapper" is the unique class slightly above price
# there is only one price tag inside this class so i will add another class inside xpath 
# Now '//text()' to get all the text inside all sub tags 

response.xpath('//div[@class="price-detailed__wrapper"]//div[@class="price"]//text()').extract()

#This expath will return a list of strings something like this

['$','2099','99']

#now join the list if 
productPrice = response.xpath('//div[@class="price-detailed__wrapper"]//div[@class="price"]//text()').extract()

productPrice = ''.join(productPrice)

#or if there is too much spaces use strip inside loop

productPrice = ''.join([x.strip() for x in productPrice])
user_109763100 
productPrice = product.xpath('//div[@class="priceformat__mainprice"]/span/text()').getall()

In this the xpath is not applied to the product but to the whole response.

In xpath when we are searching inside a xpath selector we have to use . before the xpath like this: .//div[@class="price-format__mainprice"]/span/text()

And also the price of the product is not in the page source, so it maybe generated or fetched with another request or javascript. Then you can't pull it out of this page; you will have to figure out how to fetch and extract that information separately.

user_111145840
Related