Implicit delay with Selenium

Viewed 293

I am practicing with selenium. I tried this page scraping with beautifulsoup, but as the page gets some time to load, I tried it with selenium with adding a delay time. I have tried drive.delay and timeout too. But nothing works fine with me. I get same alike error with all the tries, and I have attached my latest try and the error I get with it.

This is my ty:

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By


url = 'https://www.checkatrade.com/search?page=1&categoryId=1164&location=London'

driver =webdriver.Chrome(executable_path='D:\Web_Scraping\chromedriver_win32\chromedriver.exe')

driver.get(url)
driver.implicitly_wait(20)

items = driver.find_element_by_class_name('sc-8keyp7-0 kbPUFb sc-1a2v1qv-0 gwBNSh sc-3jeiz1-1 bfuahU')

for item in items:
    company = item.find_element_by_xpath('//*[@id="1052063"]/div/div[2]/div[2]/a/div[1]/h2').text
    print(company)

But it throws me this error, which I can not figure out.

DeprecationWarning: executable_path has been deprecated, please pass in a Service object
  driver =webdriver.Chrome(executable_path='D:\Web_Scraping\chromedriver_win32\chromedriver.exe')

DevTools listening on ws://127.0.0.1:1090/devtools/browser/9ed2753e-5d6e-4382-a9ac-a0ed2fbb8379
d:/Web_Scraping/seleniumtest.py:21: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() instead
  items = driver.find_element_by_class_name('sc-8keyp7-0 kbPUFb sc-1a2v1qv-0 gwBNSh sc-3jeiz1-1 bfuahU')
[5536:25384:1014/115806.197:ERROR:chrome_browser_main_extra_parts_metrics.cc(228)] crbug.com/1216328: Checking Bluetooth availability started. Please report if there is no report that this ends.
[5536:25384:1014/115806.198:ERROR:chrome_browser_main_extra_parts_metrics.cc(231)] crbug.com/1216328: Checking Bluetooth availability ended.
[5536:25384:1014/115806.199:ERROR:chrome_browser_main_extra_parts_metrics.cc(234)] crbug.com/1216328: Checking default browser status started. Please report if there is no report that this ends.
[5536:23016:1014/115806.200:ERROR:device_event_log_impl.cc(214)] [11:58:06.200] USB: usb_device_handle_win.cc:1048 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
[5536:25384:1014/115806.733:ERROR:chrome_browser_main_extra_parts_metrics.cc(238)] crbug.com/1216328: Checking default browser status ended.
Traceback (most recent call last):
  File "d:/Web_Scraping/seleniumtest.py", line 21, in <module>
    items = driver.find_element_by_class_name('sc-8keyp7-0 kbPUFb sc-1a2v1qv-0 gwBNSh sc-3jeiz1-1 bfuahU')
  File "C:\Python38\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 754, in find_element_by_class_name
    return self.find_element(by=By.CLASS_NAME, value=name)
  File "C:\Python38\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 1238, in find_element
    return self.execute(Command.FIND_ELEMENT, {
  File "C:\Python38\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 418, in execute
    self.error_handler.check_response(response)
  File "C:\Python38\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 243, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".sc-8keyp7-0 kbPUFb sc-1a2v1qv-0 gwBNSh sc-3jeiz1-1 bfuahU"}
  (Session info: chrome=94.0.4606.81)
Stacktrace:
Backtrace:
        Ordinal0 [0x0107BDE3+2473443]
        Ordinal0 [0x01016661+2057825]
        Ordinal0 [0x00F22438+1057848]
        Ordinal0 [0x00F4C7DB+1230811]
        Ordinal0 [0x00F76712+1402642]
        Ordinal0 [0x00F6571A+1333018]
        Ordinal0 [0x00F74ABF+1395391]
        Ordinal0 [0x00F655AB+1332651]
        Ordinal0 [0x00F42104+1188100]
        Ordinal0 [0x00F42F59+1191769]
        GetHandleVerifier [0x01202266+1549718]
        GetHandleVerifier [0x012AD4A7+2250711]
        GetHandleVerifier [0x0110718B+521403]
        GetHandleVerifier [0x01106229+517465]
        Ordinal0 [0x0101B79D+2078621]
        Ordinal0 [0x0101FB58+2095960]
        Ordinal0 [0x0101FC92+2096274]
        Ordinal0 [0x01029541+2135361]
        BaseThreadInitThunk [0x764DFA29+25]
        RtlGetAppContainerNamedObjectPath [0x77A77A9E+286]
        RtlGetAppContainerNamedObjectPath [0x77A77A6E+238]

What causes this error?

1 Answers

You are using a wrong locator.
find_element_by_class_name method is intended to receive a SINGLE class name while you are passing a multiple class names value to it "find_element_by_class_name".
To pass multiple class names you should use css selector or XPath.
So instead of

items = driver.find_element_by_class_name('sc-8keyp7-0 kbPUFb sc-1a2v1qv-0 gwBNSh sc-3jeiz1-1 bfuahU')

It could be

items = driver.find_element_by_css_selector('.sc-8keyp7-0.kbPUFb.sc-1a2v1qv-0.gwBNSh.sc-3jeiz1-1.bfuahU')

Or

items = driver.find_element_by_xpath("//div[@class='sc-8keyp7-0 kbPUFb sc-1a2v1qv-0 gwBNSh sc-3jeiz1-1 bfuahU']")
Related