Selenium on Heroku Celery Chrome Headless Crashing after multiple tasks

Question

I have set up selenium with chrome on heroku and have it set to process celery tasks. However, after doing a number of tasks (possibly tasks with bugs), it keeps crashing on any site to scrape. If I restart the celery worker, it will start working again on the same sites it just crashed on.

Here is the error:

File "/app/cheetahiq_selenium/browser.py", line 18, in chrome
'CHROMEDRIVER_PATH', ''), chrome_options=chrome_options)
File "/app/.heroku/python/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
desired_capabilities=desired_capabilities)
File "/app/.heroku/python/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/app/.heroku/python/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/app/.heroku/python/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/app/.heroku/python/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: session not created
from tab crashed
(Session info: headless chrome=80.0.3987.122)

Here are the flags I am setting:


    chrome_options = webdriver.ChromeOptions()

    prefs = {
        "download.open_pdf_in_system_reader": False,
        "download.prompt_for_download": True,
        "plugins.always_open_pdf_externally": False
    }
    chrome_options.add_experimental_option(
        "prefs", prefs
    )
    chrome_options.add_argument('--headless')
    chrome_options.add_argument('--no-sandbox')
    chrome_options.add_argument('--disable-gpu')
    chrome_options.add_argument('--window-size=1280x1696')
    chrome_options.add_argument('--hide-scrollbars')
    chrome_options.add_argument('--single-process')
    chrome_options.add_argument('--ignore-certificate-errors')
    chrome_options.add_argument('--disable-dev-shm-usage')
    if os.environ.get('ON_HEROKU', False):
        chrome_options.binary_location = os.environ.get(
            'GOOGLE_CHROME_BIN', '')
        driver = webdriver.Chrome(executable_path=os.environ.get(
            'CHROMEDRIVER_PATH', ''), chrome_options=chrome_options)
    else:
        driver = webdriver.Chrome(chrome_options=chrome_options)
    chrome_options.add_argument("--example-flag")
    driver.delete_all_cookies()
    return driver

For celery I am also setting

--concurrency=1 --max-tasks-per-child=1

so it is creating a new child for each task, but something is still getting passed to future sessions.

I am hypothesizing there is some storage the chrome driver is saving on the machine that is not being cleared each time, but I think I have set the flags to disable as much as possible. Is there anything I am missing there?

I do have the option to force restart the celery worker each time, but am looking for how to properly set up the selenium driver.

I am testing a number of flags and methods of closing of the drivers but have had no success so far (this is not reproducible easily since it only happens after processing a number of sites, or perhaps hitting some error in the driver).

Selenium on Heroku Celery Chrome Headless Crashing after multiple tasks

0 Answers