Best way to get XPATH and CSS SELECTORS for scraping with Selenium

user_11036109 0
Viewed 1134

What is the best way you guys know to get the XPATH AND CSS SELECTOR of a scraped website using selenium?

Someone suggested that I use these XPATH and CSS SELECTORS as parameters for an exercise I'm working on:

wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[placeholder='Search']"))).send_keys('Tech')
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[text()='Cancel']/.."))).click()

These parameters work very well for the exercise. However, I'm unsure on how to get (or "build") those parameters... If I use Chrome's Inspect > right click > Copy XPATH or Copy Selector, I get some very different parameters that don't seem to work as well, and don't seem to be found by selenium.

#search-bar
//*[@id="app-container"]/div/section/div/div/div[2]/button

Is there a tool or a technique to get better XPATH or CSS SELECTORS as in my first example?

web-scraping selenium xpath python
2 Answers

I like the resources shared by @JD2775. They are good to get you started understanding how to construct and understand xpaths and css selectors. When you are comfortable with that, you can work on your selector strategy. Hopefully you find at least some of the following helpful.

What makes a "good" xpath or css selector?

  • The selector should reliably and uniquely identify the targeted element.
    • For example, if an element's class occurs multiple times on the page, do not use only this class to identify the element. This is the most basic requirement for your selector
  • The selector should not be prone to "flakiness" -- ie, false failures that occur as a result of changes that are unrelated to the test.
    • Accomplish this by relying on as little of the DOM as possible to identify your element. For example, if both work to uniquely identify the element, //*[@id="app-container"]//button should be preferred over //*[@id="app-container"]/div/section/div/div/div[2]/button. Or, as you identify, "//button[text()='Cancel']/.." is the better choice.
  • Probably less important, but still worth considering: how easy is it to understand from the selector which element is being grabbed?

Some best practices

  • If you are working with a development team and thus have access to the source code of the application you are testing, implement a custom HTML attribute that is used ONLY for automation, and which has a value to uniquely identify and describe the element. In your test code you can then identify each of the elements you need with a line like this:
my_field = driver.find_element_by_css_selector('input[data-e2e-attribute="aCertainField"]')`
  • Organize your selection of elements into a Page Object Model, which abstracts the definition of webelements to one spot. So you can use these elements anywhere in your test without having to locate them, and it's easier to make changes to your selectors when necessary
user_81630710

You are correct that right-clicking and Copy Xpath is a bad way to get an Xpath. You are left with a long and brittle selector. It is much better to build your own. Once you get the hang of it, it is pretty simple to start building your own CSS and Xpath selectors. Some of them get complicated but if you keep practicing and searching for solutions you will get better and better.

The problem is it is very difficult to explain how to do it in a forum like this. Your best bet is to YouTube some videos on how to create Xpath and CSS selectors for Selenium. Here is a decent one I just found for Xpath:

https://www.youtube.com/watch?v=3uktjWgKrtI

This follows the approach I use in Chrome Dev Tools and using the built in Find window (no plugins)

Here is a good cheatsheet I have used in the past for Xpath and CSS Selectors

https://www.automatetheplanet.com/selenium-webdriver-locators-cheat-sheet/

Good luck

user_44390190
Related