Conflicting solutions for different errors with Python/Selenium/Chrome

user_1870832 0
Viewed 412

I have been getting a series of pesky Selenium/Chrome errors for a week or so, where everything works fine for a while and then suddenly doesn't. I have the right version of Chromedriver (89.0.4389.23) for my version of Google Chrome (89.0.4389.114) per Google Chrome official docs

Currently I'm getting the following error when I run driver = webdriver.Chrome(options=options) (at least on my production server, I don't get it for the same code repo on my local machine):

selenium.common.exceptions.SessionNotCreatedException: Message: session not created
from disconnected: unable to connect to renderer
  (Session info: headless chrome=89.0.4389.114)

Frustratingly, the only solution I've found for this error aside from "make sure your chrome and chromedriver match" is to remove options.add_argument("--remote-debugging-port=9222") from my webdriver options. I do have that line of code as well, and removing it appears to solve my problem for now, but my problem that line of code appears under the following comments in my codebase:

    # adding arg below fixed "DevToolsActivePort file doesn't exist."
    # ht https://github.com/heroku/heroku-buildpack-google-chrome/issues/46
    options.add_argument("--remote-debugging-port=9222")

So I've now reached a perfect circle where my Python Selenium code reliably works until it reliably breaks, with either of two different Selenium errors ("session not created from disconnected: unable to connect to renderer" or "DevToolsActivePort file doesn't exist."), where the solution to one is to add a specific line of code and the solution to the other is to remove it.

Below is my current launch_webdriver function, with comments reflecting the various transient Chrome/Selenium errors I've been trying to navigate:

from selenium import webdriver
import chromedriver_binary
from selenium.webdriver.chrome.options import Options

def launch_browser(headless=False):  # :, driverpath="./spinocchio/chromedriver"):
    options = Options()

    # if any other driver `options` are added, they must be added BELOW no-sandbox
    # per user parsecer's comment at https://stackoverflow.com/a/53073789/1870832
    # and my own excruciating experience.
    options.add_argument("--no-sandbox")

    # you may think the two lines below can be replaced with just options.headless=headless
    # please don't.
    if headless:
        options.add_argument("headless")

    # adding arg below fixed "DevToolsActivePort file doesn't exist."
    # ht https://github.com/heroku/heroku-buildpack-google-chrome/issues/46
    options.add_argument("--remote-debugging-port=9222")
    driver = webdriver.Chrome(options=options)
    return driver

How can I get to something more reliable/robust?

selenium-chromedriver web-scraping selenium google-chrome python
3 Answers

Overview

  1. Prevalent issue - per @ss7777
  2. Albeit numerous potential causes for this/related

Blue-skies and 'silver-bullet' solution I'm afraid, however, here are some common causes and potential resolve / considerations etc.

Fixes/resolve

(accompanied by subj. view of 'commonality' | frequency)

  1. Driver: ChromeDriver version/file location (~50%) - incorrect version + ensure location to executable correct location

"This issue is pretty evident when there is a incompatibility between the version of the binaries you are using." (adapted from this source, with useful summary of historical versions v/ binaries)

  1. Chrome/driver still 'active' 5-20% - despite closing down, process tree elements can remain 'alive' or 'spinning' for some time - do you notice this right after restart of PC (here)?

(Have suffered and even evidenced this previously - e.g. running code in parallel, or simply executing through .bat at same time as PyCharm console).

  1. Updates: plugin/other incompatibly (~5-15%) - noting you're using headless - does it work with head?)
  2. System: clean up your system/restore-point or worst-face (~10-20%). (restore point worked for me)
  3. Other: spawn separate thread (~5% - see here)/alternative ports (as req/feasible)

Workarounds

  • Chrome dependency (e.g. see here - albeit potential red-herring..)
  • Mode of execution (through editor main program, through CMD interface hosted via editor (e.g. PyCharm console), or through .bat file (as mentioned previously), Standalone Selenium here
  • Configuration (technical/advanced - e.g. here)for advanced users)

Unresolved (albeit with promise)

Several cases of where such a Q remains unresolved/limited contribution:

(although 2nd point here points to remedy in item 2 above).

Extreme solns:

  • Have seen intense investigation and many too-fro re Q&A - with ultimate conclusion being complete reinstallation (Chrome/python, editor) etc.

Links / related

user_136076720

I faced the same error once and I couldn't find a feasible solution anywhere. So I degraded my chrome version to v86.0
Even when you install the older version of chrome, you need to disable the chrome update to continue with your intended version. You can find the the steps here: https://www.webnots.com/7-ways-to-disable-automatic-chrome-update-in-windows-and-mac/
Along with the lower bowser version, I use the following code:

def test_setup(self):
    chrome_options=Options()
    chrome_options.add_argument('--start-maximized')
    chrome_options.add_experimental_option("excludeSwitches", ["enable-logging"])
    chrome_options.add_argument("--remote-debugging-port=9222")
    chrome_options.add_argument('--no-sandbox')
    driver=webdriver.Chrome(executable_path="//your folder path//")

Try this once and let me know if it worked for you

user_33047470

If you get this error.
Try this way:

from webdriver_manager.chrome import ChromeDriverManager
options.add_argument("--restore-last-session")
options.add_argument("--disable-gaia-services")
driver = webdriver.Chrome(ChromeDriverManager().install(),options=options)

https://peter.sh/experiments/chromium-command-line-switches/

user_148129510
Related