Selenium messes up whenever 'chromedriver' it isn't the active window

user_13034854 0
Viewed 198

(I'm working on a Macbook Pro 2020)

My selenium automation code seems to mess up whenever the chrome-driver-window isn't the 'active' window. To be more specific:

selenium needs to make a request to a website and take some data from it, but whenever I leave that chrome-driver-window minimized, it doesn't get the NEW data but instead copies what it gave me before.

I am fairly certain this isn't due to my code but because of selenium, so I want to ask: How do I allow selenium to work properly whenever the chrome-driver-window is minimized.

Here is the code:

for i in df2["keywords"]:
    time.sleep(1)
    search.send_keys(i)
    search.send_keys(Keys.RETURN)
    try:
        element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, "p.total-results.js_total_results"))
    )
    except:
        print("FIRST ELEMENT NOT FOUND")
        driver.quit()
    search = driver.find_element_by_css_selector("input#searchfor.wsp-search__input")
    data['aanbod'] = driver.find_element_by_css_selector('p.total-results.js_total_results').text
    df3 = df3.append(data, ignore_index=True)
    search.clear()
    print(data)

As you can see I'm using selenium in combination with pandas for data analysis. This is what the data looks like when I keep focus on the chrome-driver-window:

{'aanbod': '92 resultaten'}
{'aanbod': '167 resultaten'}
{'aanbod': '1.144 resultaten'}
{'aanbod': '102 resultaten'}
{'aanbod': '829 resultaten'}
{'aanbod': '91 resultaten'}
{'aanbod': '125 resultaten'}
{'aanbod': '225 resultaten'}
{'aanbod': '800 resultaten'}
{'aanbod': '276 resultaten'}

and this is what it looks like without me focussing on the window:

{'aanbod': '92 resultaten'}
{'aanbod': '92 resultaten'}
{'aanbod': '92 resultaten'}
{'aanbod': '92 resultaten'}
{'aanbod': '92 resultaten'}
{'aanbod': '92 resultaten'}
{'aanbod': '92 resultaten'}
{'aanbod': '92 resultaten'}

chrome version: mac 85.0.4183.121 (Official Build) (64-bit), chrome driver version: mac 64 ChromeDriver 85.0.4183.87

selenium python
1 Answers

If you want your script to run in the background try using the headless option for chrome. It is good practice to also add --start-maximized when working in headless mode, as there may be cases when not having this option results in elements not getting identified.

Add these lines at the beginning of your script:

from selenium.webdriver.chrome.options import Options

chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--start-maximized")

You will no longer see the browser and all the clicks, navigation etc. happening, just the results in the console.

user_121495870
Related