Selenium stuck on "Checking your browser before accessing URL"

I had the same issue with firefox. I was able to solve it by switching to Chrome.
Example code:

from selenium import webdriver
url = "<WEBSITE>"
options = webdriver.ChromeOptions()
options.add_argument("--disable-blink-features=AutomationControlled")
driver = webdriver.Chrome(options=options)
driver.get(url)

"--disable-blink-features=AutomationControlled" hides the "navigator.webdriver" flag.
See Selenium webdriver: Modifying navigator.webdriver flag to prevent selenium detection

Edit

You also have to change some default variables of chromedriver.
Example with perl:

perl -pi -e 's/cdc_/dog_/g' /path/to/chromedriver

For more details look at the original post.
See Can a website detect when you are using selenium with chromedriver?

Edit 2

Cloudflare keeps adjusting their algorithm so you could try to use undetected-chromedriver instead of the manual changing of the chromedriver.

undetected-chromedriver is an optimized Selenium Chromedriver patch which should not trigger anti-bot services. It automatically downloads the driver binary and patches it.

Wether this will work or not kinda depends on the website and the current state of the development. Cloudflare seems to track the development of undetected-chromedriver.

import undetected_chromedriver as uc
url = "<WEBSITE>"
driver= uc.Chrome()
driver.get(url)

Try using yout chrome data folder

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.utils import ChromeType

# Configure browser
options = webdriver.ChromeOptions()
options.add_argument(f"--user-data-dir=C:\\Users\\daria\\AppData\\Local\\Google\\Chrome\\User Data")
options.add_argument("--disable-blink-features=AutomationControlled")

chromedriver = ChromeDriverManager(chrome_type=ChromeType.GOOGLE, 
                                            log_level='0', 
                                            print_first_line=False).install()
driver = webdriver.Chrome(chromedriver, 
                                options=options,
                                service_log_path=None)

input ("End?")

I had the same problem when using headless Selenium on a Docker Linux image.

I solved it by creating a virtualdisplay right before calling the webdriver:

from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 800))  
display.start()

Don't forget to install both pyvirtualdisplay and xvfb: pip install pyvirtualdisplay and sudo apt-get install xvfb

And you must remove the "headless" option in ChromeDriver, here is the complete code I use :

    #Display in order to avoid CloudFare bot detection
    display = Display(visible=0, size=(800, 800))  
    display.start()
  
    options = webdriver.ChromeOptions()
    options.add_argument('--no-sandbox')
    options.add_argument('start-maximized')
    options.add_argument('enable-automation')
    options.add_argument('--disable-infobars')
    options.add_argument('--disable-dev-shm-usage')
    options.add_argument('--disable-browser-side-navigation')
    options.add_argument("--remote-debugging-port=9222")
    # options.add_argument("--headless")
    options.add_argument('--disable-gpu')
    options.add_argument("--log-level=3")
    driver = webdriver.Chrome(ChromeDriverManager().install(), chrome_options=options)

Since it was working nicely without headless on my local computer, I figured emulate a real display might do the work aswell. I do not really understand why, but from what I've understood, CloudFare tries to execute javascript code in order to confirm you're not a bot. Having a emulated webpage display helps to do so.

It is because the browser uses cloudfare to protect itself from DDOS (Distributed Denial Of Service) Attacks. There are 2 ways to solve this problem:

1. Use time.sleep -- if it takes 5 seconds for the webpage to load, just use time.sleep(5).

2. Use WebDriverWait -- for example, a button with id "sample-btn" appears only after this screen. Then what u can do is:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

btn = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, 'sample-btn'))) #Web driver waits for 10 seconds until element is visible

The 2nd one is recommended. But if the 2nd one doesn't work for u, then go with the first one. Hope that this helps!

You've run into Cloudflare's waiting room, which is the Cloudflare JavaScript challenge **or the I Am Under Attack page.

Web application firewalls (WAFs), or anti-bot services like Cloudflare, use a variety of techniques to identify you as a bot. Your client faces a number of challenges when connecting to the website, such as detecting botnets, IP reputation, HTTP request headers, TLS fingerprinting, HTTP/2 fingerprinting, event tracking and environment API tracking. Cloudflare collects the data from all the mentioned methods as sensor data and validates it for inconsistencies on the server side. Know more about Cloudflare’s bot detection from here: How does Cloudflare detect bots?

There are two different approaches to bypassing a WAF:

1. Trying to get ignored by the WAF
2. Solving the challenges and captchas

And the most straightforward option is to try to be ignored by the WAF by replicating a real browser’s properties. When you're using browser-based automation tools like Selenium, you need to copy a real browser’s properties. When you use Selenium, the document.webdriver variable is set, which can be easily identified by WAF.

You can use undetected-chromedriver. It is an optimized Selenium Chromedriver patch that should not trigger anti-bot services.

import undetected_chromedriver as uc
from webdriver_manager.chrome import ChromeDriverManager
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
import time
from fake_useragent import UserAgent
ua = UserAgent()

options = webdriver.ChromeOptions()

options.add_argument(f'user-agent={ua.random}')
chrome_path = ChromeDriverManager().install()
chrome_service = Service(chrome_path)
driver = uc.Chrome(headless = True, options=options, service=chrome_service, use_subprocess=True)

url = "https://etherdelta.com/"

driver.get(url)

time.sleep(5)

#do something
ss = driver.get_screenshot_as_png()
with open("bot-check.png", "wb") as f:
    f.write(ss)

driver.quit()

Unfortunately, even after imitating the actual browser characteristics by changing user-agent and proxies, still got stuck in the waiting room. However, the above code works if you use the non-headless Chrome version. So if you are okay with implementing it with a non-headless chrome driver, make the following changes:

. . .
driver = uc.Chrome(options=options, service=chrome_service, use_subprocess=True)
. . .

You can also use JavaScript libraries such as Puppeteer & puppeteer-extra-plugin-stealth. I tried the same approach with pyppeteer, but it didn’t work.

const puppeteer = require('puppeteer-extra');
const randUserAgent = require('rand-user-agent');
const StealthPlugin = require('puppeteer-extra-plugin-stealth')();

const agent = randUserAgent('desktop');

puppeteer.use(StealthPlugin);
puppeteer.launch({ headless: true }).then(async browser => {
  const page = await browser.newPage();

  await page.setUserAgent(agent);
  await page.goto('https://etherdelta.com/');
  await page.waitForSelector('.trading-cont');
  await page.waitForTimeout(5_000);
  await page.screenshot({ path: 'etherdelta.png', fullPage: true });
  console.log(await page.content());
  await browser.close();
});