I'm new to web scraping using python. I've coded to pull data from a job portal site using Selenium and BeautifulSoup. The flow I do is:
- Scraping the entire a link of job posting on the job portal site
- Scraping detailed info from each link of the job posting that has been obtained by looping.
I scraped the detailed info using the find_all BeautifulSoup method on the script tag type = 'application/ld+json' and data-react-helmet. But I get an error message list index out of range. Does anyone understand how to solve it?
job_main_data = pd.DataFrame()
for i, url in enumerate(URL_job_list):
headers = {
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36',
'referrer': 'https://google.com',
'Accept':
'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'en-US,en;q=0.9',
'Pragma': 'no-cache',
}
response = requests.get(url=url, headers=headers)
soup = BeautifulSoup(response.text, 'lxml')
script_tags = soup.find_all('script', attrs={'data-react helmet':'true','type':'application/ld+json'})
metadata = script_tags[-1].text
temp_dict = {}
try:
job_info_json = json.loads(metadata, strict=False)
try:
jobID = job_info_json['identifier']['value']
temp_dict['Job ID'] = jobID
print('Job ID = ' + jobID)
except AttributeError :
jobID = ''
try:
jobTitle = job_info_json['title']
temp_dict['Job Title'] = jobTitle
print('Title = ' + jobTitle)
except AttributeError :
jobTitle = ''
try:
occupationalCategory = job_info_json['occupationalCategory']
temp_dict['occupationalCategory'] = occupationalCategory
print('Occupational Category = ' + occupationalCategory)
except AttributeError :
occupationalCategory = ''
temp_dict['Job Link'] = URL_job_list
job_main_data = job_main_data.append(temp_dict, ignore_index=True)
except json.JSONDecodeError:
print("Empty response")