I was wondering ways to pull the name and current title of LinkedIn profiles using only their links with python.
Let's take this profile as example https://www.linkedin.com/in/laurence-c-osborn-636281ba/:

I want to capture the info of their name Laurence C. Osborn and their description in this case Attorney at Baker, Keener & Nahra LLP. For this I have copied their paths using both XpathFinder and Inspect
Name:
- /html/body/div[6]/div[3]/div/div/div[2]/div/div/main/section1/div[2]/div[2]/div1/div1/h1
- //*[@id="ember31"]/div[2]/div[2]/div1/div1/h1
Description:
- /html/body/div[6]/div[3]/div/div/div[2]/div/div/main/section1/div[2]/div[2]/div1/div[2]
- //*[@id="ember31"]/div[2]/div[2]/div1/div[2]
I have watched many tutoriales about webscrapping using LinkedIn, most of them require to connect to an account, but i want to do it without entering to an account. Which would be the best way to achieve this?
This is what I've been trying so far from bs4 import BeautifulSoup from lxml import etree import requests
URL = "https://www.linkedin.com/in/laurence-c-osborn-636281ba/"
HEADERS = ({'User-Agent':
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 \
(KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36',\
'Accept-Language': 'en-US, en;q=0.5'})
webpage = requests.get(URL, headers=HEADERS)
soup = BeautifulSoup(webpage.content, "html.parser")
dom = etree.HTML(str(soup))
print(dom.xpath('//*[@id="ember31"]/div[2]/div[2]/div[1]/div[1]/h1')[0])
The thing is that I don't get any output just []