Can't use Selenium to find elements. Try to avoid deprecated commands warning, got result only one element. (on Colab)

user_17470235 0
Viewed 218

As my title said. To avoid deprecated warning of Selenium.I try to use the new format as I found. and somehow result return only 1 element. How can I fix it to return elements? FYI I scraping website. I'm working on Colab so I can share with my workmates easier.

Here my code

#!pip install selenium
#!pip install IPython
#!pip install pandas
#!apt-get update
#!apt install chromium-chromedriver

from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome('chromedriver',chrome_options=chrome_options)

import pandas as pd
from IPython.display import display
from selenium.webdriver.common.by import By
website = 'https://www.bitkub.com/fee/cryptocurrency'
driver.get(website)

#pull_data
coins = driver.find_elements(By.TAG_NAME,'tr')

checking what element it contains (which somehow got only 1)

print(coins)
[<selenium.webdriver.remote.webelement.WebElement (session="091bac0ea46529043f4d84786ebc705f", element="76396e66-6516-4dd0-9d85-bc66bfad4676")>]
selenium-webdriver web-scraping selenium python
2 Answers

What is occuring is that you need webdriver waits and wait for visibility for all elements.

wait = WebDriverWait(driver, 10)
trs = wait.until(ec.visibility_of_all_elements_located((By.TAG_NAME, "tr")))

Import:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec
user_99012610

Ideally coins should have been a list of elements.

However, to extract the name of the coins you need to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

  • Using CSS_SELECTOR:

    driver.get('https://www.bitkub.com/fee/cryptocurrency')
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "tbody tr td:nth-child(2) span")))])

  • Using XPATH:

    driver.get('https://www.bitkub.com/fee/cryptocurrency')
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//following::td[2]//span")))])

  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
user_74294470
Related