I'm trying to open a page to web scrape using headless Chrome, but I am getting a self-XSS warning and the page is not retrieved.
This error occurs at the "get" stage, before I actually attempt to execute any scripts on the page:
[0527/071700.434:INFO:CONSOLE(505)] "%c%s color: red; background: yellow; font-size: 24px; WARNING!", source: /_/mss/boq-identity/_/js/k=boq-identity.ConsentUi.en_GB.VucJXlAkXyo.es5.O/am=CwAQ/d=1/excm=_b,_tp,mainview/ed=1/dg=0/wt=2/rs=AOaEmlECz_87YTuhKKwEvkQJQGH1ukQd8w/m=_b,_tp (505)
[0527/071700.434:INFO:CONSOLE(505)] "%c%s font-size: 18px; Using this console may allow attackers to impersonate you and steal your information using an attack called Self-XSS.
Do not enter or paste code that you don't understand.", source: /_/mss/boq-identity/_/js/k=boq-identity.ConsentUi.en_GB.VucJXlAkXyo.es5.O/am=CwAQ/d=1/excm=_b,_tp,mainview/ed=1/dg=0/wt=2/rs=AOaEmlECz_87YTuhKKwEvkQJQGH1ukQd8w/m=_b,_tp (505)
The page opens fine if I use webdriver to open the GUI.
I have tried using the flags that other guides and answers suggest for avoiding this (many suggest --disable-xss-auditor), but this has not helped. I've tried setting the executable path manually rather than using ChromeDriverManager as well, but that didn't change anything either.
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--disable-xss-auditor")
driver = webdriver.Chrome(executable_path=ChromeDriverManager().install(),options=chrome_options)
time.sleep(1)
driver.get(page_url)