Python Selenium Chrome not working on Google Cloud Run Jobs

Question

I am attempting to use Google Cloud Run jobs to host a scraping script that uses Python Selenium to periodically scrape a certain amount of information from a job board site. The script is working locally with Selenium running fine, but when I try to get it to work on Cloud Run Jobs, I am consistently getting the error message:

{'message': 'Task #0, Attempt #0 failed: No chrome executable found on PATH', 'severity': 'ERROR'}

Printing out the system path using sys.path gives this:

PATH is: ['/workspace', '/layers/google.python.runtime/python/lib/python310.zip', '/layers/google.python.runtime/python/lib/python3.10', '/layers/google.python.runtime/python/lib/python3.10/lib-dynload', '/layers/google.python.pip/pip/lib/python3.10/site-packages', '/layers/google.python.runtime/python/lib/python3.10/site-packages']

I initialize the Selenium drivers using the following: driver = webdriver.Chrome(service = Service(ChromeDriverManager().install()), options=options)

I have searched for similar errors, but it seems that most of the time people in this situation are seeing a missing chromedriver error, as opposed to a missing chrome executable.

I am admittedly very new to Docker and containerization, so my hunch is that this has something to do with the Dockerfile. I have looked at other stackoverflow articles with similar issues and have tried to implement those fixes in my Dockerfile, but to no avail. The posts are below:

I took the chromedriver and chrome installation section of the Dockerfile from this help article: https://nander.cc/using-selenium-within-a-docker-container

My Dockerfile currently looks like this:

FROM python:3.9

COPY . ./

# Install manually all the missing libraries
RUN apt-get update
RUN apt-get install -y gconf-service libasound2 libatk1.0-0 libcairo2 libcups2 libfontconfig1 libgdk-pixbuf2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libxss1 fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils default-jdk

# install google chrome
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list'
RUN apt-get -y update
RUN apt-get install -y google-chrome-stable

# install chromedriver
RUN apt-get install -yqq unzip
RUN wget -O /tmp/chromedriver.zip http://chromedriver.storage.googleapis.com/`curl -sS chromedriver.storage.googleapis.com/LATEST_RELEASE`/chromedriver_linux64.zip
RUN unzip /tmp/chromedriver.zip chromedriver -d /usr/local/bin/

# set display port to avoid crash
ENV DISPLAY=:99

RUN pip install --no-cache-dir -r requirements.txt

CMD [ "python", "./main.py" ]

What could be causing this issue and what steps can I take to avoid this "No chrome executable found on PATH" error?

Python Selenium Chrome not working on Google Cloud Run Jobs

0 Answers