Nothing is printed or code does not execute when I have scrapped data

user_10272555 0
Viewed 26

I have been trying to scrape some data off pnet job site however the code does print the results of the scraped data. I run the file but nothing happens the code does not finish executing I have waited over an hour and the code has not executed. I don't know what the solution is

from bs4 import BeautifulSoup
import requests


html_text = requests.get('https://www.pnet.co.za/5/job-search-detailed.html? 
what=Data%20analysis&where=Durban&radius=30&searchOrigin=Resultlist_top- 
search&whatType=skillAutosuggest').text
print(html_text)
soup = BeautifulSoup(html_text, 'lxml')`enter code here`
web-scraping python
1 Answers

You should add headers. An example is given below.

It's also better to add a timeout value, though this should run without one.

headers = {"User-Agent": "Chrome/100.0.4896.127"}

response = requests.get('https://www.pnet.co.za/5/job-search-detailed.html?what=Data%20analysis&where=Durban&radius=30&searchOrigin=Resultlist_top-search&whatType=skillAutosuggest', headers=headers, timeout=10)

print(response.text)
user_52053930
Related