python selenium scroll down slowly in div container

user_15274265 0
Viewed 58

I'm scraping a page that I cannot share because it's private and needs access. However, I have a box in the page which is a div container that I need to scroll down in order to get all HTML loaded. I tried many ways but scrolling down is still fast and is not waiting for all elements to load. So I need a way to slow down scrolling in a div container

My code so far:

dialog = driver.find_element_by_xpath("//div[@id='search-results-container']")
driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", dialog)

This is working and scrolling down to the bottom of the page but it's fast. I need to slow it down

Any idea?

web-scraping python
2 Answers
user_183173910

You can try something like this

while True:
    driver.execute_script(
        """
        container = document.querySelector("<div container css selector>")
        container.scrollBy(0,10)
        """
        )
    # sleep time if it takes long to load(probably won't be needed)
    # stop condition

This way the div container will scroll slowly (10px) until the stop condition is met

user_163762640
Related