How to scrape more than one page of critic reviews from Rotten Tomatoes?

user_7914054 0
Viewed 499

I've been using this scraper to scrape critic reviews for this URL: https://www.rottentomatoes.com/m/avengers_endgame/reviews Although, I've been struggling with how to go through additional pages as this currently scrapes critic reviews of the first page. Does anyone know how I would go about this?

import selenium
from selenium import webdriver
import pandas as pd
driver = webdriver.Chrome()
driver.get("https://www.rottentomatoes.com/m/avengers_endgame/reviews")
review_1df = pd.DataFrame(columns=['Date', 'Reviewer', 'Website', 'Review', 'Score'])
dates = []
reviews = []
scores = []
newscores = []
names = []
sites = []
results = driver.find_elements_by_class_name("review_area")
reviewnum = 1
reviewers = driver.find_elements_by_class_name("col-xs-8")

for r in results:
    dates.append(r.find_element_by_class_name('subtle').text)
    reviews.append(r.find_element_by_class_name('the_review').text)
    revs = r.find_element_by_class_name('review_desc')
    scores.append(revs.find_element_by_class_name('subtle').text)
    
    for r in reviewers:
        names.append(r.find_element_by_xpath('//*[@id="reviews"]/div[2]/div[4]/div[' +str(reviewnum)+ ']/div[1]/div[3]/a[1]').text)
        sites.append(r.find_element_by_xpath('//*[@id="reviews"]/div[2]/div[4]/div[' +str(reviewnum)+']/div[1]/div[3]/a[2]/em').text)
        reviewnum+=1

for score in scores:
    if score == ('Full Review'):
        newscores.append('no score')
    else:
        score2 = score[14:]
        newscores.append(score2)
        
review_1df['Date'] = dates
review_1df['Review'] = reviews
review_1df['Score'] = newscores
review_1df['Reviewer'] = names
review_1df['Website'] = sites
scrape web-scraping python
1 Answers

You can use URL parameters to get to the next page of reviews and repeat the same steps. For example, the following url will take you to the second page of reviews:

https://www.rottentomatoes.com/m/avengers_endgame/reviews?type=&sort=&page=2

Note the parameters are type=&sort=&page=2 where you can also specify the sorting and type. Change it to page=3 to get to the third page.

You'll also have to add a check to see if the page even exists. For example, you'll get no reviews on this URL:

https://www.rottentomatoes.com/m/avengers_endgame/reviews?type=&sort=&page=200000
user_135976750
Related