Finding web elements with selenium only working with .driver and not other variables

user_16357019 0
Viewed 41

e.g.

This is the code used to scrape title and views from a specific channel.

videos = driver.find_elements_by_class_name("style-scope ytd-grid-renderer")

from selenium import webdriver
driver.get("url")

driver.implicitly_wait(10)

videos = driver.find_elements_by_class_name("style-scope ytd-grid-renderer")

for video in videos: 
    title = videos.find_elements_by_xpath('.//*[@id="video-title"]')
    views = videos.find_elements_by_xpath('.//*[@id="metadata-line"]/span[1]')
    item = {
        'title' : title, 
        'views' : views
    }

Error displayed in terminal:

title = videos.find_elements_by_xpath('.//*[@id="video-title"]')
AttributeError: 'list' object has no attribute 'find_elements_by_xpath'

When using findElement() function in selenium, it is only valid when its driver.find_elements_by_xpath, and not when driver is replaced with a variable. (videos.find_elements_by_xpath)

Bit of a beginner, so there might be concepts I'm missing here. I am aware of [this post] and the solution provided, but it's solution did not seem to work and was quite confusing.

If there is a concept that should be learned to solve this problem, please some resources or explanations if you can. Thank you very much

web-scraping selenium python
1 Answers

This error

AttributeError: 'list' object has no attribute 'find_elements_by_xpath'

is cause, this line

title = videos.find_elements_by_xpath('.//*[@id="video-title"]')

should have been

title = video.find_elements_by_xpath('.//*[@id="video-title"]')

videos is a list in your code and video is a web element.

A list does not have find_elements or find_element function.

But a webelement has both the mentioned function.

so changing to video should get the job done.

for video in videos: 
    title = video.find_elements_by_xpath('.//*[@id="video-title"]')
    views = video.find_elements_by_xpath('.//*[@id="metadata-line"]/span[1]')
    item = {
        'title' : title, 
        'views' : views
    }
user_94440730
Related