Here is the link where you can find the Google reviews of a listing I am trying to scrape. I have pasted the code below to get all the reviews, not more than 1200. When my code reached to scrape more than 500 reviews, then the following error happened
selenium.common.exceptions.WebDriverException: Message: unknown error: session deleted because of page crash from unknown error: cannot determine loading status from tab crashed
Why is it happening? Is it because the selenium does not support lots of data handling when scrolling more than 500 reviews?
all_reviews_div = self.driver.find_element_by_class_name("review-dialog-list"
)
while True:
self.driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", all_reviews_div)
all_reviews = all_reviews_div.find_elements_by_class_name('gws-localreviews__google-review')
sys.stdout.write("\r" + "Scrolling Reviews " + str(len(all_reviews))+ ' of '+ str(self.total_reviews))
sys.stdout.flush()
if(len(all_reviews) == self.total_reviews or len(all_reviews) >= 1200):
break