I try to scrape sales navigator linkedin and when I scrape I need to scrool to the bottom of the page for it to get fully loaded but I cannot do it because it consists of two subpages. I tried to solve it by locating item at the leads subpage and use selenium click object but it's still not working. How to solve it?
profile_url = "https://www.linkedin.com/sales/search/people?query=(recentSearchParam%3A(doLogHistory%3Atrue)%2Cfilters%3AList((type%3AREGION%2Cvalues%3AList((id%3A102393603%2Ctext%3AAsia%2CselectionType%3AINCLUDED)))))&sessionId=fvwzsZ8CTAKdGri5T5mYZw%3D%3D"
driver.get(profile_url)
time.sleep(20)
to_scroll = driver.find_element("id", "search-results-container")
to_scroll.click()
start = time.time()
initialScroll = 0
finalScroll = 1000
while True:
driver.execute_script(f"window.scrollTo({initialScroll},{finalScroll})")
initialScroll = finalScroll
finalScroll += 1000
time.sleep(5)
end = time.time()
if round(end - start) > 30:
break
time.sleep(5)
How can I get to the bottom if this page?