Web scraping with python, javascript output

user_16674073 0
Viewed 96

I am trying to scrap the job information from this website and have been stuck for a few days. When I print the soup.text output I get a short javascript text which is not what I want as I want the html element. I have seen similar solutions to implement 'Header less browsing' but when I implemented that I just received several errors. I am new to web-scraping and have looked at various tutorials, videos and simply am not getting the output I want and have no idea what I am doing wrong.

import requests
from bs4 import BeautifulSoup



def aSwiftScraper():

    jobLinks = []
    pages = []
    URL = "https://www.amiqus.com/jobs?options=,20993,20877,20876&page=1"
    page = requests.get(URL)
    soup = BeautifulSoup(page.content, "html.parser")
    print(soup.text)


aSwiftScraper()
python-requests beautifulsoup html python
1 Answers

Try to change User-Agent HTTP header when making request to the server:

import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
}

url = "https://www.amiqus.com/jobs?options=,20993,20877,20876&page=1"

soup = BeautifulSoup(requests.get(url, headers=headers).content, "html.parser")
for title in soup.select(".attrax-vacancy-tile__title"):
    print(title.get_text(strip=True))

Prints:

Engine Programmer C++ AAA opportunity - Remote working
Senior Programmer
Gameplay Programmer

...
user_100359850
Related