Firefox and Seleniumwire not working on AWS EC2 instance

Viewed 258

I'm trying to run Firefox with Seleniumwire on an AWS EC2 instance. Here's my current configuration:

  • ubuntu==20.4
  • selenium-wire==4.3.0
  • python==3.8.10
  • geckodriver==0.29.1
  • ChromeDriver==94.0.4606.81
  • 64-bit (Arm)

I've installed ChromeDriver with sudo apt-get install chromium-chromedriver and geckodriver with sudo apt-get install firefox-geckodriver.

Here's the script I'm trying to run:

from seleniumwire import webdriver

options = webdriver.FirefoxOptions()
options.add_argument("--headless")
options.add_argument('--no-sandbox')
options.add_argument('--ignore-certificate-errors-spki-list')
options.add_argument('--ignore-ssl-errors')
options.add_argument("--window-size=1920,1080")
options.add_argument("--disable-gpu")

d = webdriver.Firefox(options=options)
d.get("https://google.com")
print(d.requests)

The problem: d.requests returns [], which is unexpected behavior. I know there's been issues posted about driver.requests returning an empty list, but the recommended solutions haven't worked on my setup. Also print(d.page_source) works fine, which leads me to think there's some incorrect proxy configuration. Also, I isolated this problem to Firefox, since the same code works on Chrome (with Firefox swapped out for Chrome); that is, d.requests contains a non-empty list of requests. I would use Chrome, but the scope of my project requires Firefox.

Here're some steps I've taken to debug:

  • Setting the port explicitly using selenium_options={"port": 4444}, where 4444 is taken from the localhost output of geckodriver on the command line.
  • Adding a proxy using the following format:
options = {
        'proxy': {
            'http': 'http://myusername:password@myproxyserver.com:123456', 
            'https': 'http://myusername:password@myproxyserver.com:123456',
            'no_proxy': 'localhost,127.0.0.1'
        }  
    }
  • Re-installing and re-running all my scripts on a new EC2 instance.
  • Specifying a different tmp directory, per this issue

I have yet to try different versions of geckodriver and selenium-wire, but I suspect that's not the issue.

Any suggestions on debugging would be greatly appreciated.

0 Answers
Related