Why doesn't an element appear even after waiting for the element to load in Selenium?

user_12276162 0
Viewed 93

I have a specific URL I want to scrape, however, a certain element doesn't load until the page is fully loaded and I am having trouble getting Selenium to find it:

from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

driver = webdriver.Chrome(Insert Driver Path Here)
game_url = "https://rbsdm.com/stats/box_scores/?_inputs_&type=%22reg%22&away=%22ARI%22&home=%22SF%22&year=2020"
driver.get(game_url)
element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, "table table-striped table-hover table-responsive"))
    )
print(element)

The <div> that has the table I want has the ID qbs, which loads, however, it doesn't contain anything upon initial load, apparently. The table has the class name I use when making the element variable.

The page's data is definitely loaded after the 10-second window I give it, so I'm confused about this isn't working.

web-scraping selenium google-chrome beautifulsoup python
1 Answers

In your code, when you are searching for the desired HTML element with:

element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, "table table-striped table-hover table-responsive"))
    )

You are looking for a class name, but table table-striped table-hover table-responsive isn't a class name, it's a CSS selector (with chaining the spaces with a dot .):

table.table-striped.table-hover.table-responsive

So, either you use the above CSS selector (notice By.CSS_SELECTOR):

element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, "table.table-striped.table-hover.table-responsive"))
    )

Or use the class name:

element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, "table"))
    )

Here's a complete working example:

from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait


driver = webdriver.Chrome()
game_url = "https://rbsdm.com/stats/box_scores/?_inputs_&type=%22reg%22&away=%22ARI%22&home=%22SF%22&year=2020"
driver.get(game_url)
element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, "table"))
    )
print(element.text)

Output:

Player Plays EPA/play Total EPA aDoT CMP% xCMP% CPOE xQBR
K.Murray 54 0.18 9.9 5.2 70.3 72.5 -2.2 63.6
J.Garoppolo 38 0.1 3.6 7 59.4 65.6 -6.2 54.9
user_123497340
Related