How to set path to chromedriver in Google Cloud Composer

user_11111158 0
Viewed 1009

I am attempting to run a DAG that will use selenium to scrape a web page every week on Cloud Composer.

I have already tried to give the path when creating the WebDriver.Chrome() instance to a driver that I uploaded to GCS, though I imagine this is not the best way to do this.

Airflow is giving this error

Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home

If you have any tips as to updating Cloud Composer's PATH variable, would be greatly appreciated. If I need to put in more info, drop a comment and I'll add on.

google-cloud-composer selenium-chromedriver selenium python-3.x path
1 Answers

So there was no official answer and the Slack channel for neither Composer nor GKE were able to help. The real problem was that the binaries were not on Composer. Best answer for right now is to manually ssh into all of your GKE airflow-workers and install Google Chrome yourself: https://linuxize.com/post/how-to-install-google-chrome-web-browser-on-ubuntu-18-04/

Then place the chromedriver for the correct version of Chrome you installed in your dags/dependencies folder and reference it on instantiation of your Webdriver object. Hope this helps!

user_111111580
Related