Disclaimer -- I am new to python!
For my first project I am learning to scrape indeed.com hiring data using bs4. The data I want to extract is the job title, which is located in the span of two different h2's (because it seems that the job title for the 'new' jobs are categorised differently from the others).
The source data is from this url: https://uk.indeed.com/jobs?q=devops&l=United+Kingdom&start=0
Here is my code:
import requests
from bs4 import BeautifulSoup
#extract
def extract(page):
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Safari/605.1.15'}
url = f'https://uk.indeed.com/jobs?q=devops&l=United+Kingdom&start={page}'
r = requests.get(url, headers)
soup = BeautifulSoup(r.content, 'html.parser')
return soup
#transform
def transform(soup):
title1 = soup.find_all('h2', class_ = 'jobTitle jobTitle-color-purple')
title2 = soup.find_all('h2', class_ = 'new topLeft')
for item in title1:
title = item.find('span').text
print(title)
for item in title2:
title = item.find('span').text
print(title)
c = extract(0)
transform(c)
Although my code runs fine, my question is: is it possible for me to rewrite this code so I only have one for loop that checks the data for title1 and title2? I'd like to learn good coding practice so any help refactoring this is much appreciated!