Pull name and description from LinkedIn profiles using python

Viewed 33

I was wondering ways to pull the name and current title of LinkedIn profiles using only their links with python.

Let's take this profile as example https://www.linkedin.com/in/laurence-c-osborn-636281ba/: Example LinkedIn profile

I want to capture the info of their name Laurence C. Osborn and their description in this case Attorney at Baker, Keener & Nahra LLP. For this I have copied their paths using both XpathFinder and Inspect

Name:

  1. /html/body/div[6]/div[3]/div/div/div[2]/div/div/main/section1/div[2]/div[2]/div1/div1/h1
  2. //*[@id="ember31"]/div[2]/div[2]/div1/div1/h1

Description:

  1. /html/body/div[6]/div[3]/div/div/div[2]/div/div/main/section1/div[2]/div[2]/div1/div[2]
  2. //*[@id="ember31"]/div[2]/div[2]/div1/div[2]

I have watched many tutoriales about webscrapping using LinkedIn, most of them require to connect to an account, but i want to do it without entering to an account. Which would be the best way to achieve this?

This is what I've been trying so far from bs4 import BeautifulSoup from lxml import etree import requests

URL = "https://www.linkedin.com/in/laurence-c-osborn-636281ba/"
  
HEADERS = ({'User-Agent':
            'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 \
            (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36',\
            'Accept-Language': 'en-US, en;q=0.5'})
  
webpage = requests.get(URL, headers=HEADERS)
soup = BeautifulSoup(webpage.content, "html.parser")
dom = etree.HTML(str(soup))
print(dom.xpath('//*[@id="ember31"]/div[2]/div[2]/div[1]/div[1]/h1')[0])

The thing is that I don't get any output just []

0 Answers
Related