Find <span> in multiple h2 using beautifulsoup

Viewed 313

Disclaimer -- I am new to python!

For my first project I am learning to scrape indeed.com hiring data using bs4. The data I want to extract is the job title, which is located in the span of two different h2's (because it seems that the job title for the 'new' jobs are categorised differently from the others).

The source data is from this url: https://uk.indeed.com/jobs?q=devops&l=United+Kingdom&start=0

Here is my code:

import requests
from bs4 import BeautifulSoup

#extract

def extract(page):
    headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Safari/605.1.15'}
    url = f'https://uk.indeed.com/jobs?q=devops&l=United+Kingdom&start={page}'
    r = requests.get(url, headers)
    soup = BeautifulSoup(r.content, 'html.parser')
    
    return soup

#transform

def transform(soup):
    title1 = soup.find_all('h2', class_ = 'jobTitle jobTitle-color-purple')
    title2 = soup.find_all('h2', class_ = 'new topLeft')
    
    for item in title1:
        title = item.find('span').text
        print(title)
    
    for item in title2:
        title = item.find('span').text
        print(title)
    
c = extract(0)
transform(c)

Although my code runs fine, my question is: is it possible for me to rewrite this code so I only have one for loop that checks the data for title1 and title2? I'd like to learn good coding practice so any help refactoring this is much appreciated!

1 Answers

If I understand you correctly, you want to search only for job titles that have new label on top of them:

def transform(soup):
    titles = soup.select("h2 .new + span")
    for title in titles:
        print(title.text)

Prints:

DevOps Engineer
Senior DevOps Engineer
DevOps Engineer
DevOps Engineer
Senior DevOps Engineer - Fully Remote
DevOps Engineer
Software Engineer (PHP or Golang)
DevOps Engineer

h2 .new + span means:

select all <span> tags which are following tags with class="new" under <h2> tags.

Related