Scroll to bottom of infinity scrolling page

user_5404647 0
Viewed 431

I have already read this answer, however, I need to scroll the webpage in steps, because if I scroll it directly to the height of the documents the images are not loaded, only a placeholder in base64url. I would like to scroll by a little until I get to the end of the webpage, making sure that I can see in the driver the images, so they are loaded.

Any way to achieve this?

My current function:

def scroll_to_bottom(driver):
    last_height = driver.execute_script("return document.body.scrollHeight")
    while True:
        # Scroll down to bottom
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        # Wait to load page
        time.sleep(2)
        # Calculate new scroll height and compare with last scroll height
        new_height = driver.execute_script("return document.body.scrollHeight")
        if new_height == last_height:
            break
        last_height = new_height

This goes directly to te bottom, but if the item is not focused for a elapsed time it won't load the images I need.

selenium python
3 Answers

You could simulate a mouse scroll by inserting the code bellow to your loop.

from pynput.mouse import Controller

mouse = Controller()
mouse.scroll(0 , -50)

Make sure you import pynput library first:

pip3 install pynput

Watch this short video for more mouse simulations.

user_144883540

You can try like below:

1: Use of ActionChains:

from selenium.webdriver.common.action_chains import ActionChains

driver.get("https://shop1236011695.v.weidian.com/?userid=1236011695&spider_token=70ce")

action = ActionChains(driver)

# Create two variables, one for calucuting height before scroll and one for after scroll
height = 0
newheight = 1

while height != newheight: # Keep scrolling until the end
    height = driver.execute_script("return document.body.scrollHeight")
    action.send_keys(Keys.END).perform()
    time.sleep(2)
    newheight = driver.execute_script("return document.body.scrollHeight")
    print(height,newheight)
print("FINISHED SCROLLING")

794 3235
3235 8866
8866 21226
21226 21891
21891 21891
FINISHED SCROLLING

2: Locate the products and apply scrollIntoView:

i= 0
try:
    while True:
        options = driver.find_elements_by_xpath("//div[@id='components-wrapper']/div[2]/div")
        driver.execute_script("arguments[0].scrollIntoView(true);",options[i])
        i+=1
        time.sleep(2)
except Exception as e:
    print(e)
    pass

Update : You can also use Keys.DOWN

height = 0

while height < 40000: # Keep scrolling until the end
    height += 1000
    action.send_keys(Keys.DOWN).perform()
    time.sleep(2)
    print(height)
print("FINISHED SCROLLING")
user_164528400

You can achieve this with below code

document.evaluate('container identifier', document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue.scrollBy(0,pixel size to scroll)

it will scroll only given pixels size at a time, then you can put wait or condition to check that element is loaded or not and then continue. also you can change pixels size using variable so you can scroll till desired location. please modify the last argument as per your need. Also Please replace container identifier with actual identifier.

Please try below code

def scroll_to_bottom(driver):
        last_height = driver.execute_script("return document.body.scrollHeight")
        while True:
            # Scroll down to 100 pixels each time
            self.driver.execute_script("document.evaluate('container identifier', document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue.scrollBy(0,100)")
            # Wait to load page
            time.sleep(2)
            # Calculate new scroll height and compare with last scroll height
            new_height = driver.execute_script("return document.body.scrollHeight")
            if new_height == last_height:
                break
            last_height = new_height
user_85571960
Related