Headless webdriver returns error but non-headless works

user_460056 0
Viewed 2247

I am doing a simple experiment with Amazon and Webdriver. However, using Webdriver Headless cannot find elements and errors out, but non-headless works.

Any suggestions how to get it working headless? There is a comment right above the --headless flag.

from selenium import webdriver
import sys
import os

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


def get_inventory(url):

    chrome_options = Options()

    # Making it headless breaks. Commenting
    # this line, making it non-headless works.  
    chrome_options.add_argument("--headless")

    chrome_options.add_experimental_option(
        "prefs", {'profile.managed_default_content_settings.javascript': 2})
    chrome_options.binary_location = '/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary'

    driver = webdriver.Chrome(executable_path=os.path.abspath(
        'chromedriver'), chrome_options=chrome_options)

    driver.set_window_size(1200, 1000)

    try:
        driver.get(url)

        add_to_cart_button_xp = '//*[@id="add-to-cart-button"]'
        add_to_cart_button = driver.find_element_by_xpath(add_to_cart_button_xp)
        add_to_cart_button.click()

        driver.get('https://www.amazon.com/gp/cart/view.html/ref=lh_cart')

        qty_field_xp = '//div/input[starts-with(@name, "quantity.") and @type="text"]'
        qty_field = driver.find_element_by_xpath(qty_field_xp)
        qty_field.clear()
        qty_field.send_keys("2")

        update_link_xp = f'//input[@value="Update" and @type="submit"]'
        update_link = driver.find_element_by_xpath(update_link_xp)
        update_link.click()



url = 'https://www.amazon.com/Pexio-Professional-Stainless-Food-Safe-Dishwasher/dp/B07BGBSY9F'
get_inventory(url)
selenium-webdriver python
3 Answers

I think you just had some selector issues. I checked the elements and updated the quantity setting; everything else should be pretty much the same, aside from the binary locations.

from selenium import webdriver
import sys
import os

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


def get_inventory(url):
    chrome_options = Options()

    chrome_options.add_argument("--headless")

    driver = webdriver.Chrome(
        executable_path='/usr/bin/chromedriver', 
        chrome_options=chrome_options,
    )
    chrome_options.add_experimental_option(
        "prefs", 
        {'profile.managed_default_content_settings.javascript': 2},
    )
    chrome_options.binary_location = '/usr/bin'

    driver.set_window_size(1200, 1000)

    try:
        driver.get(url)
        driver.save_screenshot("/tmp/x1.png")

        driver.find_element_by_xpath('//*[@id="add-to-cart-button"]').click()

        driver.get('https://www.amazon.com/gp/cart/view.html/ref=lh_cart')

        driver.find_element_by_xpath("//span[@data-action='a-dropdown-button']").click()
        driver.find_element_by_xpath("//*[@class='a-dropdown-link'][text()[contains(., '2')]]").click()

        driver.find_element_by_class_name("nav-logo-base").click()

        driver.save_screenshot("/tmp/confirm.png")
        driver.close()
    except Exception as e:
        print(e)


url = 'https://www.amazon.com/Pexio-Professional-Stainless-Food-Safe-Dishwasher/dp/B07BGBSY9F'
get_inventory(url)

I've run this with and without --headless and it's working fine for me. I navigated to the homepage at the end so you can confirm the quantity change worked (hence the screenshot).

user_8338810

What is the behavior you see?

When I enabled headless, scripts started failing because running headless slows execution down.

I currently run chrome with these options:

'--no-sandbox', '--headless', '--window-size=1920,1080', '--proxy-server="direct://"', '--proxy-bypass-list=*'

The last two options supposedly help with the slowness, but I didn't see any difference.

Hope this helps.

user_9737180

I verified your claim on my Mac (using /Applications/Google Chrome.app/Contents/MacOS/Google Chrome).

My guess is that, since you are moving from an item page to the cart page of Amazon, the cookies are lost, so that the cart page won't show any item, and therefore won't contain any text input with a name starting with “quantity”, which is what the exception is about.

Googling for headless chrome cookies yields this page, which in turn points to this page, the content of which could also be about your problem. Be it this, or be it a particularly smart behavior of the Amazon website, the fact remains: the cookie that stores the cart (or a key thereof, but the result is the same) is not read by the cart page when in headless mode.

user_10460070
Related