Getting 403 when using Selenium to automate checkout process

Question

I am trying to create a script using python and selenium to automate the checkout process at bestbuy.ca.

I get all the way to the final stage where you click to review the final order, but get the following 403 forbidden message (as seen in the network response) when I try to click through to the final step.

Is there something server side that has detected that I am using selenium and preventing me to proceed?

How can I hide the fact that it is selenium being used?

These are the options I am using for selenium:

options = Options()
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_argument("start-maximized")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(options=options)

I currently have 10 second delays after each action(ie open page, wait, click add to cart, wait, click checkout, wait)

I have implemented a random useragent to be used on each run:

import fake_useragent
ua = UserAgent()
userAgent = ua.random
options.add_argument(f'user-agent={userAgent}')

I have also modified my chromedriver binary as per the comments in THIS THREAD

Error seen when proceeding to order review page:

user_1807561 · Answer

After much testing the last few days, here are the options that have allowed me to bypass the restrictions I was facing.

Modified cdc_ string in my chromedriver

Chromedriver options:

 options.add_argument('--disable-blink-features=AutomationControlled')
 options.add_argument("--disable-extensions")
 options.add_experimental_option('useAutomationExtension', False)
 options.add_experimental_option("excludeSwitches", ["enable-automation"])
 chrome_driver = webdriver.Chrome(options=options)

Change the property value of the navigator for webdriver to undefined:

chrome_driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")

After all three of these were implemented I no longer faced any 403 error when navigating the site and the cart/checkout process.

user_3228768 · Answer

In my case, either using code to control the browser, or simply starting Chrome through python and manually using the browser always leads to the 403 error, even just adding a product to the cart.

As you said, I think that this site someway knows that the user is using Selenium or some sort of automation tool and the server is blocking API requests.

Searching in stackoverflow I found this https://stackoverflow.com/a/52108199/3228768 but editing the chromedriver results anyway in a failure.

The only way I completed the flow is settings this options:

u = 'https://www.bestbuy.ca/en-ca/category/appliances/26517'
# relevant part start here
options = webdriver.ChromeOptions()
options.add_argument("--disable-blink-features")
options.add_argument("--disable-blink-features=AutomationControlled")
# relevant part ends here 
driver = webdriver.Chrome(executable_path=r"chromedriver.exe", options=options)
driver.maximize_window()
driver.get(u)

In this way I managed to add a product to the cart. I think you could use it to proceed the flow until checkout. Let me know.

user_8068153 · Answer

Try this one: https://github.com/ultrafunkamsterdam/undetected-chromedriver

It avoids the selenium detection quite well, I've been having good result with it so far. Headless is not guaranteed though.

import undetected_chromedriver as uc
driver = uc.Chrome()
driver.get('https://www.bestbuy.ca/')

Getting 403 when using Selenium to automate checkout process

3 Answers