I'm trying to use Selenium with Firefox on Databricks but can't get it to work, and I haven't come across any others with quite the same issues as me. The problem seems to lie with the Firefox installation.
My first approach was to install Selenium, geckodriver and Firefox within the notebook. The first two installed fine, but the Firefox installation showed some network connection issues. Here's what I ran:
- Install Selenium
%pip install selenium
- Install geckodriver
%sh
wget https://github.com/mozilla/geckodriver/releases/download/v0.31.0/geckodriver-v0.31.0-linux64.tar.gz -O /tmp/geckodriver.tar.gz
%sh
tar -xvzf /tmp/geckodriver.tar.gz -C /tmp
- Install Firefox
%sh
/usr/bin/yes | sudo apt-get update --fix-missing
which gives the error
Ign:1 https://apt.datadoghq.com stable InRelease
Err:2 https://apt.datadoghq.com stable Release Could not handshake: Error in the pull function. [IP: xx.xxx.xx.xx xxx]
Err:3 http://security.ubuntu.com/ubuntu focal-security InRelease Connection failed [IP: xxx.xxx.xxx.xx xx]
Hit:4 https://repos.azul.com/zulu/deb stable InRelease
Err:5 http://archive.ubuntu.com/ubuntu focal InRelease Connection failed [IP: xx.xxx.xx.xx xxx]
Err:6 http://archive.ubuntu.com/ubuntu focal-updates InRelease Connection failed [IP: xxx.xxx.xxx.xx xx]
Err:7 http://archive.ubuntu.com/ubuntu focal-backports InRelease Connection failed [IP: xx.xxx.xx.xx xxx]
I tried skipping the apt-get update command and running
%sh
sudo apt-get --yes --force-yes install firefox
but it resulted in similar errors.
I later discovered that our databricks environment is set up in such a way that we can't send anything outside of our network, so this approach of installing Firefox will not work.
My second approach was to download and install Firefox to my local machine and upload the firefox.exe file to the dbfs. I installed selenium and the geckodriver as above and ran the following code.
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
options = Options()
options.headless = True
options.binary_location = r'/dbfs/path/to/firefox/firefox.exe'
driver = webdriver.Firefox(options=options, executable_path = '/path/to/driver/geckodriver')
which gave the error
driver = webdriver.Firefox(options=options, executable_path = '/path/to/driver/geckodriver')
InvalidArgumentException: Message: binary is not a Firefox executable
It seems that for some reason, it can't read the firefox.exe file that I've uploaded to dbfs.
Any suggestions as to possible fixes for this? Or any alternative ways to get Selenium working with Firefox (taking the network restrictions in to account)?