XPath expression fails to return correct elements in Selenium (Java)

Viewed 58

I am trying to collect a list of products in Amazon. Specifically, I am going to the following URL: https://www.amazon.com/dp/[ASIN]/ref=olp-opf-redir?aod=1&ie=UTF8&condition=ALL where [ASIN] is the unique Amazon Standard Identification Number for the item in question. For this issue, assume the URL to be for these airpods: https://www.amazon.com/dp/B09JQMJHXY/ref=olp-opf-redir?aod=1&ie=UTF8&condition=ALL

Notice that this URL opens a side panel with a listing of different vendors selling the item in different conditions (i.e. new, used, used like new, etc.).

I created an XPath expression to obtain some of these items. The basic XPath for this is //div[@id='aod-offer-list']/div[@id='aod-offer']. I further refined this to return a list of items that are shipped only from Amazon:

//div[@id='aod-offer-list']/div[@id='aod-offer' and div[@id='aod-offer-shipsFrom']/div/div/div/span[text()='Amazon']]

When I evaluate this expression in Chrome, I get the list of offers I am interested in. However, when I run this from Eclipse, I get a list of offers that consist of multiple copies of the pinned offer at the very top of the side panel. The bizarre thing is that the pinned offer (//div[@id='aod-pinned-offer']) is not even a child of the offer list (//div[@id='aod-offer-list']). In fact, the pinned offer and the offer list are siblings of each other. Given these facts, how is it that I am getting a different WebElement list when executing in Java than when evaluating the same XPath directly in Chrome.

The relevant code:

public static void main(String[] args) {

    System.setProperty("webdriver.chrome.driver", "C:/Program Files/WebDrivers/chromedriver.exe");
    WebDriver driver = new ChromeDriver();
    driver.get("https://www.amazon.com/dp/B09JQMJHXY/ref=olp-opf-redir?aod=1&ie=UTF8&condition=ALL");
    
    List<WebElement> offers = new ArrayList<>();
    try {
//      merchants = driver.findElements(By.xpath(xpath));
        new WebDriverWait(driver, Duration.ofSeconds(10)).until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//div[@id='aod-offer-list']")));
        String xpath = "//div[@id='aod-offer-list']/div[@id='aod-offer' and div[@id='aod-offer-shipsFrom']/div/div/div/span[text()='Amazon']]";
        offers = new WebDriverWait(driver, Duration.ofSeconds(10)).until(ExpectedConditions.presenceOfAllElementsLocatedBy(By.xpath(xpath)));
        System.out.println("Found " + offers.size() + " offers.");
        Iterator<WebElement> iter = offers.iterator();
        while (iter.hasNext()) {
            String script = "return arguments[0].innerHTML";
            WebElement offer = iter.next();

            WebElement soldByElement = offer.findElement(By.xpath("//a[@aria-label='Opens a new page']"));
            String soldByText = (String) ((JavascriptExecutor) driver).executeScript(script, soldByElement);
            System.out.println("Sold by: " + soldByText);

            WebElement priceElement = offer.findElement(By.xpath("//span[@class='a-offscreen']"));
            String priceString = (String) ((JavascriptExecutor) driver).executeScript(script, priceElement);
            System.out.println("Price for item " + priceString);
        }
    } catch (TimeoutException toe) {
        System.err.println(toe);
    }
    driver.quit();
}

The output:

Found 4 offers.
Sold by:  Adorama
Price for item $174.00
Sold by:  Adorama
Price for item $174.00
Sold by:  Adorama
Price for item $174.00
Sold by:  Adorama
Price for item $174.00

The output should've been:

Found 2 offers.
Sold by:  Amazon Warehouse
Price for item $160.08
Sold by:  Amazon Warehouse
Price for item $165.30

The incorrect output is pulling the price from the pinned item and the "Sold By" value from one of the vendors not shipping from Amazon. My unproven theory is that the relative path to the "Sold By" and "Price" elements are not relative from the offer element, but from the DOM itself. I tried adding a dot (.) to the XPath string, but that is not a correct notation. I need to force Selenium to resolve the path starting from the obtained offer element.

UPDATE:

If I add the following snippet

String script = "return arguments[0].innerHTML";
WebElement offer = iter.next();
String offerElement = (String) ((JavascriptExecutor) driver).executeScript(script, offer);
System.out.println(offerElement);

it prints out the correct "innerHTML" for the list of offers. In other words, I can see all the correct elements if I use this Xpath

String xpath = "//div[@id='aod-offer-list']/div[@id='aod-offer']";
0 Answers
Related