How to write the css/xpath selectors for expedia page (nested HTML) with scrapy python

user_14202689 0
Viewed 110

I would like to scrape data points highlighted in the picture (hotel names, location, reviews, ratings, and prices) but my spider is not returning anything (most likely due to wrong selectors). The URL to the website is here:

enter image description here

Here is my spider code:

class ExpediaSpider(scrapy.Spider):
    name = 'expedia'
    # allowed_domains = ['expedia.com']
    start_urls = [all_urls[0]]

    def parse(self, response):
        items = ExpediaScraperItem()

        html = response.css('.uitk-card-link')

        for qoutes in html:

            review = qoutes.css('div.listing__reviews all-t-margin-two').css('::text').extract()
            price = qoutes.css('span.uitk-cell loyalty-display-price all-cell-shrink').css('::text').extract()
            hotel_name = qoutes.css('truncate-lines-2 all-b-padding-half pwa-theme--grey-900 uitk-type-heading-500').css('::text').extract()
            location = qoutes.css('overflow-wrap uitk-spacing uitk-spacing-padding-blockend-two uitk-text-secondary-theme').css('::text').extract()

            # then save it
            items['review'] = review  # eqauls to var extracted
            items['price'] = price
            items['hotel_name'] = hotel_name
            items['location'] = location

         
            yield items

I also tried directly listing the selectors without a loop but I am striking out. If anyone has some time and could explain to me some CSS/XPath tricks to this HTML "blob" that would be awesome. Thank you for taking the time to read this.

scrapy web-scraping css-selectors xpath python
1 Answers

This will work:

hotels = response.xpath('//li[@data-stid="property-listing"]')

for hotel in hotels:
    review = hotel.xpath('string(.//div[@data-stid="content-hotel-review-info"]/span/span[1])').get()
    price = hotel.xpath('.//span[@data-stid="price-lockup-text"]/text()').get()
    hotel_name = hotel.xpath('.//h3[@data-stid="content-hotel-title"]/text()').get()
    location = hotel.xpath('.//div[@data-test-id="content-hotel-neighborhood"]/text()').get()
    
    # then save it
    items['review'] = review  # eqauls to var extracted
    items['price'] = price
    items['hotel_name'] = hotel_name
    items['location'] = location
    
    yield items
user_164297800
Related