I get a problem when I try to scrape to the bottom of the page selenium

Viewed 24

I try to scrape sales navigator linkedin and when I scrape I need to scrool to the bottom of the page for it to get fully loaded but I cannot do it because it consists of two subpages. I tried to solve it by locating item at the leads subpage and use selenium click object but it's still not working. How to solve it?

profile_url = "https://www.linkedin.com/sales/search/people?query=(recentSearchParam%3A(doLogHistory%3Atrue)%2Cfilters%3AList((type%3AREGION%2Cvalues%3AList((id%3A102393603%2Ctext%3AAsia%2CselectionType%3AINCLUDED)))))&sessionId=fvwzsZ8CTAKdGri5T5mYZw%3D%3D"

driver.get(profile_url)

time.sleep(20)

to_scroll = driver.find_element("id", "search-results-container")

to_scroll.click()

start = time.time()

initialScroll = 0
finalScroll = 1000

while True:
    driver.execute_script(f"window.scrollTo({initialScroll},{finalScroll})")
    initialScroll = finalScroll
    finalScroll += 1000


    time.sleep(5)

    end = time.time()

    if round(end - start) > 30:
        break
time.sleep(5)

How can I get to the bottom if this page?

0 Answers
Related