I'm trying to use Selenium to scrape data off of betonline.ag and when I switch to using a headless browser (need to do this to eventually move script to AWS lambda function) I receive the following error:
File "C:\Users\adelu\anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 246, in check_response
raise exception_class(message, screen, stacktrace, alert_text) # type: ignore[call-arg] # mypy is not smart enough here
selenium.common.exceptions.UnexpectedAlertPresentException: Alert Text: There is no admin configuration!
Message: unexpected alert open: {Alert text : There is no admin configuration!}
(Session info: headless chrome=105.0.5195.127)
Stacktrace:
Backtrace:
Ordinal0 [0x006BDF13+2219795]
Ordinal0 [0x00652841+1779777]
Ordinal0 [0x0056423D+803389]
Ordinal0 [0x005BEFFE+1175550]
Ordinal0 [0x005AE616+1107478]
Ordinal0 [0x00587F89+950153]
Ordinal0 [0x00588F56+954198]
GetHandleVerifier [0x009B2CB2+3040210]
GetHandleVerifier [0x009A2BB4+2974420]
GetHandleVerifier [0x00756A0A+565546]
GetHandleVerifier [0x00755680+560544]
Ordinal0 [0x00659A5C+1808988]
Ordinal0 [0x0065E3A8+1827752]
Ordinal0 [0x0065E495+1827989]
Ordinal0 [0x006680A4+1867940]
BaseThreadInitThunk [0x77296739+25]
RtlGetFullPathName_UEx [0x77828FD2+1218]
RtlGetFullPathName_UEx [0x77828F9D+1165]
Checking one of the div's, I can see that this is a CloudFlare issue:
driver.find_elements(By.TAG_NAME, 'div')[2].text
'Oops! It’s likely that we’re having\nan internal systems issue\nIf the problem persists, please contact
customer support so we can help.\nAccess denied\nYou cannot access ultraplay.betonline.ag. Refresh the
page or contact the site owner to request access.\nRay ID: 74d6c9316f273b7c\nTimestamp: 2022-09-20
01:28:42 UTC\nYour IP address: ##.###.##.###\nRequested URL: ultraplay.betonline.ag/esports/early-
markets\nError reference number: 1020\nServer ID: FL_106F56\nUser-Agent: Mozilla/5.0 (Windows NT 10.0;
Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.5195.127 Safari/537.36\nCloudflare Ray
ID:74d6c9316f273b7c\nPerformance & Security by Cloudflare\nHelp Center'
I'm currently using undetected_chromedriver because I was receiving the same error prior to doing this and after I switched to the undetected driver it fixed the issue. Once I tried using a headless browser it started giving me this error and haven't been able to get around it.
Here is my current code:
from math import floor
from datetime import datetime
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager
import pandas as pd
import undetected_chromedriver as uc
import os
import time
EARLY_MARKETS = 'https://ultraplay.betonline.ag/esports/early-markets'
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument("start-maximized")
options.add_argument("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/96.0.4664.110 Safari/537.36")
driver = uc.Chrome(options=options)
driver.get(EARLY_MARKETS)