Alert me when Job is available using BeautifulSoup

user_13142580 0
Viewed 150

I am trying to create an alert for when a specific job listing is available at the company Northvolt (https://northvolt.com/career). The title of the position is "Energy coordinator". At the moment they are not looking for my position so I want to create a script that can alert me when it's available.

I have tried to use BeautifulSoup. The idea is to scrape/search the career webpage every day for the word "Energy coordinator" and send an email to me whenever it's available.

The problem is that I don't think that I receive the right HTML in my terminal when I request it. I am not sure how to describe it or if any of this makes sense. Anyway below is some code.

    import requests
    from bs4 import BeautifulSoup
    
    result = requests.get("https://northvolt.com/career")
    
    source = result.text
    
    soup = BeautifulSoup(source, "html.parser")
    print(soup.prettify())

When I print the soup and search for various keywords from the webpage in the terminal search bar I don't find the keywords. When I use another webpage in requests.get("") I find everything in the terminal that is displayed on the webpage. What am I doing wrong here, why is my request from Northvolt's webpage so weird compared to other webpages?

This image shows what I receive in my terminal. A wall of text which doesn't say anything about job listings

This image shows what I receive in my terminal. A wall of text which doesn't say anything about job listnings

alert beautifulsoup python
1 Answers

The jobs are fetched using AJAX on that page. If you open the "Network" tab in your browser, you should see one of the requests fetch JSON data from https://feed.jobylon.com/feeds/150d2f7a9df44df8a1ac9c5c794f7220/?format=json.

I'm assuming the ID in that URL is the company's ID on Jobylon.

BeautifulSoup only fetches static HTML pages. It doesn't load them and execute JavaScript like browsers do. Selenium is capable of doing that though.

However, in this case you can just fetch the JSON data directly from the URL above in your Python code.

For example:

import requests

url = "https://feed.jobylon.com/feeds/150d2f7a9df44df8a1ac9c5c794f7220/"

querystring = {"format":"json"}

headers = {
    'Accept': "application/json",
    'Accept-Language': "en-US,en;q=0.5",
    'Content-Type': "application/json;charset=UTF-8",
    'Origin': "https://northvolt.com",
    'DNT': "1",
    'Referer': "https://northvolt.com/career",
    'Connection': "keep-alive",
    'TE': "Trailers",
    'cache-control': "no-cache",
    }

response = requests.request("GET", url, headers=headers, params=querystring)

data = response.json()

print(data)

To generate Python code like the above easily I usually right-click a request in Firefox's Network tab, click on 'Copy as cURL', import it into Postman (by clicking on "Import" and then on "Paste Raw Text") and then click on "Code" under the request in Postman and export it as a Python code snippet.

I uploaded a short demo video here.

user_34866750
Related