How to scroll down to the last comment of a youtube video

user97Vb1 0
Viewed 41

I am learning web scraping using selenium. In a task to scrape all the comments on a YouTube video I need to scroll down to the last comment. I have tried the following code but it's not working -

# imports used

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
import time


driver = webdriver.Chrome()
url = 'https://www.youtube.com/watch?v=etzmAZ7oiz0'

driver.get(url)
time.sleep(3)

last_height = driver.execute_script("return document.body.scrollHeight")
html = driver.find_element(By.TAG_NAME, 'html')

while True:
    # Scroll down to bottom
    html.send_keys(Keys.PAGE_DOWN)

   # Wait to load the page
    time.sleep(5)


    --- Code to scrape comments ---


   # Calculate new scroll height and compare with last scroll height

    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    else:
        last_height = new_height

Will be grateful for any help

data-extraction selenium-webdriver web-scraping selenium python
1 Answers

Get the locator of the last comment and build the webelement, pass it on to the below function, where 'args' is the Webelement built using the last comment's locator(css/xpath etc). Below method will scroll to the Webelement which you will pass to in the place of 'args'

driver.execute_script("arguments[0].scrollIntoView()", args)
Nischitha K N0
Related