How to wait for the page to fully load?

Question

I try to webscrape linkedin sales navigator but the page loads other elements only if i scroll down and wait a bit. I tried to execute this a following way but it just added me 3 more elements so I get 6 out of 25.

profile_url = "https://www.linkedin.com/sales/search/people?query=(recentSearchParam%3A(doLogHistory%3Atrue)%2Cfilters%3AList((type%3AREGION%2Cvalues%3AList((id%3A102393603%2Ctext%3AAsia%2CselectionType%3AINCLUDED)))))&sessionId=fvwzsZ8CTAKdGri5T5mYZw%3D%3D"

driver.get(profile_url)

time.sleep(20)

action = webdriver.ActionChains(driver)
to_scroll = driver.find_element(By.XPATH, "//button[@id='ember105']")
to_scroll_up = driver.find_element(By.XPATH, "//button[@id='ember141']")
action.move_to_element(to_scroll)
action.perform()
time.sleep(3)
action.move_to_element(to_scroll_up)
action.perform()
time.sleep(3)
action.move_to_element(to_scroll)
action.perform()
time.sleep(3)

How to solve it?

user2MRa1 · Answer

You can do this by using Explicit Wait. As it will make sure the element is visible on UI. You can identify an element that gets loaded after the page is completely loaded and replace XPATH_of_element with its XPATH. Also, add this code before extracting the data so it works properly.

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

WebDriverWait(driver, 15).until(EC.visibility_of_element_located((By.XPATH, 'XPATH_of_element')))

You can visit this blog by Shreya Bose for more information.

How to wait for the page to fully load?

1 Answers