Python Selenium webdriver: Does not scroll down the page if browser is minimised

user_2293224 0
Viewed 563

I wrote a script using Python Selenium Webdriver to scrape the app reviewers from the google play store. I wrote a script in such a way that it scrolls down the page of user reviewer and click on 'show more reviews' button 5 times. After 5 times, it checks whether the last review date was less than a given date then it stopped scrolling otherwise it keeps scrolling. Here is the code:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import time
import datetime as dt
from datetime import datetime

driver = webdriver.Chrome()
baseurl = 'https://play.google.com/store/apps/details?id=com.mapmyrun.android2&showAllReviews=true'
driver.get(baseurl)

WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"//div[./span[text()='Most relevant']]"))).click()
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"//div[@role='option'][./span[text()='Newest']]"))).click() 

counter = 0
while True:
    driver.execute_script("window.scrollTo(0,document.body.scrollHeight)")
    time.sleep(2)
    counter = counter + 1
    if len(driver.find_elements_by_xpath("//span[text()='Show More']"))>0:
        driver.find_element_by_xpath("//span[contains(text(),'Show More')]").click()
        counter = 0
    if counter == 10:
        
        person_info = driver.find_elements_by_xpath("//div[@class='d15Mdf bAhLNe']")
        last_date = person_info[-1].find_element_by_xpath(".//span[@class='p2TkOb']").text
        print(datetime.strptime(last_date,"%B %d, %Y").strftime("%Y/%m/%d"))
        if datetime.strptime(last_date,"%B %d, %Y").strftime("%Y/%m/%d") < dt.datetime(year=2020,month=12,day=2).strftime("%Y/%m/%d"):
            break;
        
        else:
            counter = 0


   print(counter)

Now the above code works fine if Chrome windows remain active on the screen. However, if I minimize the chrome browser, then it keeps showing the same old date again and again. For example, as the script started running, I minimized the chrome browser, after the count of 5 it shows the date 2021/02/07. As the date is greater than 2020/12/2, the loop will continue and counter is reset. However, the second time it again shows the same date (i.e. 2021/02/07), it keeps repeating until I maximized the chrome browser again. Is there any way in selenium webdriver that it keep scrolling down the pages even the page is minimzed or inactive?

selenium-webdriver selenium python-3.x python
1 Answers

selenium is a remote control for the browser as if a human is using the browser. the actions that selenium do is meant to mimic the human actions as close as possible. for example selenium refuses to click a button that exists on the page but is not visible for the human. of course since selenium is a program and not a human it can only estimate what actions are possible for a human and sometimes makes mistakes.

in your case: if you minimize the browser practically no human actions are possible. that you are getting results at all is basically a mistake of selenium. ideally you should get an error message stating no action possible if browser minimized. so the result that you are getting is a side effect of a mistake that just coincidentally results in the old date.

the solution for your problem is quite easy: since you don't want to see the browser anyway you should use headless mode. headless mode means the browser is opened but not drawn to screen. it is still drawn to some hidden buffer. so the website (and selenium) thinks the browser is open normaly (not minimized). but it is not visible for the human using the computer.

the code looks somethings like this:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.headless = True
driver = webdriver.Chrome(options=chrome_options)
user_3608990
Related