Selenium w/ Firefox 88 changes to navigator.webdriver

user_9593068 0
Viewed 2035

The most recent release of Firefox 88 included a bug fix that ended a user's ability to set the navigator.webdriver property to false if the browser is being run via the marionette protocol. Unfortunately, this effectively renders some web scraping activities impossible if a site uses a service like Cloudflare's DDOS protection which checks that property on the remote browser and gets caught in an endless loop rather than redirecting to the requested page.

Is anyone aware of an alternative approach now that methods like

profile.set_preference("dom.webdriver.enabled", False)

or

profile.set_preference("useAutomationExtension", False)

now have no effect?

Specifically, I'm using Selenium with Python bindings but I imagine a workaround using any framework would be translatable to my situation.

selenium-webdriver web-scraping selenium firefox python
2 Answers

I tried adding extension to Firefox which overrides navigator.webdriver before any other script is loaded. It seems like it has something to do not only with navigator.webdriver but with something else.

I executed the following code on document_start

script = document.createElement("script"); script.type = 'text/javascript'; script.id = "scripttttttt"; script.innerHTML = "Object.defineProperty(navigator, 'webdriver', {get: () => undefined});document.getElementById('scripttttttt').remove();"; document.documentElement.prepend(script);

This is enough to open https://ebay-kleinanzeigen.de/m-einloggen.html but not enough to log into Google Account.

user_158339800

You can stop firefox updating to version 88. For that perform below tasks.

  1. Remove firefox version 88 if already updated/installed.

    sudo apt-get purge firefox

  2. Remove firefox profile (If still exists).

    rm -rf ~/.mozilla/

  3. Search firefox older version from cache.

    apt-cache show firefox | grep Version

  4. Install the desired version other than 88.

    sudo apt-get install firefox=desiredversion

    For example:

    sudo apt-get install firefox=75.0+build3-0ubuntu1

  5. Create a file policies.json with below policies in location: /usr/lib/firefox/distribution. This will stop force update to version 88.

    {
"policies": {
    "AppUpdateURL": "http://127.0.0.1/",
    "DisableAppUpdate": true,
    "DisableFirefoxStudies": true,
    "DisableSystemAddonUpdate": true,
    "DisableTelemetry": true,
    "ExtensionUpdate": false
}

    }

  6. Open firefox goto about:config and click on I accept the risk! or whatever it shows.

  7. Search for app.update.auto and toggle it to False.

  8. Finally check version to confirm.

user_119933090
Related