Incomplete HTML-response on some sites using Requests & BeautifulSoup or Selenium

user_14968980 0
Viewed 197

I'm tying to scrape information from some urls using Requests and BeautifulSoup in Python. But some sites only return an partial HTML response missing the content of the page

This is the code, that is not working:

import requests
from bs4 import BeautifulSoup
url = "http://www.exampleurl.com"
r = requests.get(url)
soup = BeautifulSoup(r.content, 'html.parser')

Here is the incomplete response: Picture

I tried to use Selenium with Chrome Webdriver instead, but ended up with the same issue.

from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')
options.add_argument('--headless')
browser = webdriver.Chrome(options=options)
browser.get(url)
html = browser.page_source

Any ideas?

python-requests selenium beautifulsoup python
1 Answers

What happens

  1. You do not get the expected html cause it is in an iframe
  2. Try to get the src of the iframe soup.find('iframe')['src'] and request with it again.

Example

import requests
from bs4 import BeautifulSoup
url = "http://www.ingenieur-jobs.de/jobangebote/3075/"
r = requests.get(url)
soup = BeautifulSoup(r.content, 'html.parser')

iframe = requests.get(soup.find('iframe')['src'])

soup = BeautifulSoup(iframe.content, 'html.parser')
soup
user_144608240
Related