I want to scrape the reviews for https://www.sephora.com/product/double-wear-stay-in-place-makeup-P378284?icid2=products%20grid:p378284 But the xpath I found is not returning any value:
response.xpath('//*[@id="ratings-reviews"]/div[5]/div[1]/div[2]/div[1]/div[4]/text()').extract()
any help is appreciated.
You need to implement a separate request to extract Reviews as these are not available in the page source. Try this
def start_requests(self):
url = 'https://api.bazaarvoice.com/data/reviews.json?Filter=contentlocale%3Aen*&Filter=ProductId%3AP378284&Sort=SubmissionTime%3Adesc&Limit=30&Offset=0&Include=Products%2CComments&Stats=Reviews&passkey=rwbw526r2e7spptqd2qzbkp7&apiversion=5.4&Locale=en_US'
return [Request(url=url, callback=self.parse)]
def parse(self, response):
review = json.loads(response.text)['Results']
In the review variable you will have all the review's related fields like ReviewText, Rating
As stated by @Mads Hansen, you can't scrape this without Javascript support. So yes, Selenium or similar tool would be required here. Alternatively, you can download the JSON containing the reviews and extract the comments from there :
https://api.bazaarvoice.com/data/reviews.json?Filter=contentlocale%3Aen*&Filter=ProductId%3AP378284&Sort=SubmissionTime%3Adesc&Limit=30&Offset=0&Include=Products%2CComments&Stats=Reviews&passkey=rwbw526r2e7spptqd2qzbkp7&apiversion=5.4&Locale=en_US
Code for the text reviews :
import requests
resp = requests.get("https://api.bazaarvoice.com/data/reviews.json?Filter=contentlocale%3Aen*&Filter=ProductId%3AP378284&Sort=SubmissionTime%3Adesc&Limit=30&Offset=0&Include=Products%2CComments&Stats=Reviews&passkey=rwbw526r2e7spptqd2qzbkp7&apiversion=5.4&Locale=en_US")
data = resp.json()
for el in data['Results']:
print (el['ReviewText'])
Output :
