Selenium script captures first few items out of 100

Viewed 700

I've rceated a script to scrape different collection names from a webpage traversing multiple pages. The script can parse first 13 names from each page out of 100 names. One such collection name looks like Pudgy Penguins. How can I capture 100 names instead of only 13 from different pages of that site using selenium?

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

link = "https://opensea.io/rankings"

def scroll_to_the_bottom():
    check_height = driver.execute_script("return document.body.scrollHeight;")
    while True:
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        try:
            WebDriverWait(driver,5).until(lambda driver: driver.execute_script("return document.body.scrollHeight;")  > check_height)
            check_height = driver.execute_script("return document.body.scrollHeight;") 
        except TimeoutException:
             break


def get_collection_names(link):
    driver.get(link)

    while True:
        scroll_to_the_bottom()
        for item in WebDriverWait(driver,10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"[role='listitem'] [class$='Ranking--row']"))):
            collection_name = WebDriverWait(item,10).until(EC.visibility_of_element_located((By.CSS_SELECTOR,"[class$='Ranking--collection-name']"))).text
            yield collection_name

        try:
            button = WebDriverWait(driver,10).until(EC.presence_of_element_located((By.XPATH,"//button[contains(@class,'Buttonreact__StyledButton')][./i[contains(.,'arrow_forward')]]")))
            driver.execute_script('arguments[0].click();',button)
            WebDriverWait(driver,10).until(EC.staleness_of(item))
        except Exception as e:
            return

if __name__ == '__main__':
    driver = webdriver.Chrome()
    for collection_name in get_collection_names(link):
        print(collection_name)

Scrolling to the bottom of every page seems not to have any effect on the number of results the script produces.

2 Answers

I have checked your description and source code and I think there are many elements. So it doesn't load at one time. For solving this, scroll down to the bottom step by step. So, I have changed function scroll_to_the_bottom as below :

def scroll_to_the_bottom() :
    H = driver.execute_script('return document.body.scrollHeight;')
    h = 0
    while True :
        h += 300
        if h >= H :
            break
        
        driver.execute_script("window.scrollTo({}, {});".format(0, h))
        time.sleep(1)

So, embed above code into your code, we can change it as below :

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


link = "https://opensea.io/rankings"


def get_collection_names(link):
    driver.get(link)
    unique_items = set()
    while True:
        item = WebDriverWait(driver,10).until(EC.presence_of_element_located((By.CSS_SELECTOR,"[class$='Ranking--collection-name']")))
        H = driver.execute_script('return document.body.scrollHeight;')
        h = 0
        while True :
            h += 300
            if h >= H :
                break

            for element in WebDriverWait(driver,10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,"[class$='Ranking--collection-name']"))):
                if element.text not in unique_items:
                    yield element.text
                unique_items.add(element.text)
            driver.execute_script("window.scrollTo(0, {});".format(h))
            time.sleep(1)

        try:
            button = WebDriverWait(driver,10).until(EC.presence_of_element_located((By.XPATH,"//button[contains(@class,'Buttonreact__StyledButton')][./i[contains(.,'arrow_forward')]]")))
            driver.execute_script('arguments[0].click();',button)
            WebDriverWait(driver,10).until(EC.staleness_of(item))
        except Exception as e:
            return


if __name__ == '__main__':
    driver = webdriver.Chrome()
    for item in get_collection_names(link):
        print(item)
    driver.quit()

Hope to be helpful for you. Thanks.

The problem seems to be that the content is dynamically created.

I can see two possible solutions for this problem:

  1. Very simply just start your broswer in a ridiculous resolution like 1920x10000 and you will force the page to create all the 100 elements you are looking for.

    driver.set_window_size(1920, 10000)
    

    If you execute the following commands before and after you resize the window you can observe that, at HD resolution (1920, 1080) it will dinamically produce 22 elements. After you resize it to 1920x10000 it will give you all 100 items. Bare in mind that I've done this with the geckodriver/Firefox webdriver for selenium, if I remember correctly, chromedriver is bugged on this implementation and will not resize the window. (Maybe they fixed it, don't know)

  2. You can scroll through every element you yield and the next one will be generated automatically, but this solution is more complex (even if it's the smartest choice for long and durable crawlers/scrapers on dynamic websites like this one or instagram etc.).

    First you need to find all elements created at the first cycle (so you are going to store them). Next you can yield the result and store the unique data captured (so next time you are going to check for new elements dynamically created, you are not gonna store the same ones). Once there are no new elements you can click the next page button or either kill the browser if you are done.

    Example:

    lets first set a unique list of elements:

    u_nfts = list() #lets say that what makes them unique is their name, so we are going to store that in this list
    

    now the part where we store the nft names and check if there are new ones dynamically created.

    nft_elements = driver.find_elements_by_xpath("//div[@role='listitem']")
    for element in nft_elements:
        #here you have to extract the data you need, don't copy paste this code, you need to implement it+
        nft_name = element.get_attribute("name") #there is no such attribute, lets just use it as a dummy
        u_nfts.append(nft_name)
    
    # scroll to the last element located so the browser can generate new elements if there is any
    driver.execute_script("arguments[0].scrollIntoView();", nft_elements[-1])
    

    Now what happens is that next time you are going to ask for nft_elements (by saying driver.find_elements_by_xpath("//div[@role='listitem']") you can check if the elements you acquired are already stored in u_nfts. If there are no new elements you can break the cycle or click the next button and do all over again.

Hope this was helpful.

Related