I get a problem when scraping linkedin sales navigator

Viewed 37

I try to scrape leads from linkedin sales navigator using selenium and bs4 and get a problem that on the page I try to scrape there's 25 leads and I got only 3 leads in the result of scraping. How to fix it?

from selenium import webdriver
from bs4 import BeautifulSoup
import time
from selenium.webdriver.common.by import By

driver = webdriver.Chrome("C:/Users/ospre/PycharmProjects/Linkedin parser/Driver/chromedriver.exe")

driver.get("https://linkedin.com/uas/login")

time.sleep(5)

username = driver.find_element("id", "username")

username.send_keys("email")

pword = driver.find_element("id", "password")

pword.send_keys("password")

driver.find_element(By.XPATH, "//button[@type='submit']").click()

profile_url = "https://www.linkedin.com/sales/search/people?query=(recentSearchParam%3A(doLogHistory%3Atrue)%2Cfilters%3AList((type%3AREGION%2Cvalues%3AList((id%3A102393603%2Ctext%3AAsia%2CselectionType%3AINCLUDED)))))&sessionId=fvwzsZ8CTAKdGri5T5mYZw%3D%3D"

driver.get(profile_url)
start = time.time()

# will be used in the while loop
initialScroll = 0
finalScroll = 1000

start = time.time()

# will be used in the while loop
initialScroll = 0
finalScroll = 1000

while True:
    driver.execute_script(f"window.scrollTo({initialScroll},{finalScroll})")

    initialScroll = finalScroll
    finalScroll += 1000


    time.sleep(25)

    end = time.time()

    if round(end - start) > 165:
        break

src = driver.page_source

soup = BeautifulSoup(src, 'lxml')

intro = soup.find_all('a', {'data-anonymize': 'person-name'})

name = []

for name_object in intro:
    name.append(name_object.text.strip())

print(name)

And in the result I get only 3 lead like ['Aloki Batra', 'Girish Jhunjhnuwala', 'Parikshith Arya']. I think that is the problem with page loading, I scroll down to the end of the page, it helped a bit and I got 9 leads. But when I tried to increase I got only 15 leads when there's 25 some of them got just skipped. How to fix this?as

0 Answers
Related