Unable to parse elements from a website that contains tabular data (iframe)

We are trying to parse href attributes from the DOM of a job website. We want to get an href for each job.

We usually use CSS paths and pass those to Selenium's find_elements_by_css method.

Unfortunately, we've noticed that the browser plugin SelectorGadget had trouble providing us with a CSS path. We proceeded to use a CSS path using Google Chrome (ctrl+shift+c). Chrome could extract a path, but neither Selenium nor BeautifulSoup can work with those paths.

After many failed attempts to extract the elements using different classes and tags, we believe something is entirely wrong with either our approach or the website. We hypothesize that the desired elements are impossible to parse by Selenium and BeautifulSoup for whatever reason? Could the iframe tags in the DOM be a source of error (see this SO question)? What makes the parsing fail here, and is there a way to get around this problem? A website-related problem source would also explain why the SelectorGadget was unable to get a path in the first place. Our conclusion would be to use regular expressions to extract the href attributes that we need. This would only be a last resort solution.

For German-speakers, please note that there is a spelling error in the target elements: <div class="stellenagebot">. Please do not let yourself get confused by those (as we did).

No luck with BeautifulSoup:

import re
import requests
from bs4 import BeautifulSoup


url = 'https://www.artemiskliniken.de/Karriere/%C3%84rzte/'

page = requests.get(url)

soup = BeautifulSoup(page.text, 'html.parser')

print(soup.prettify())
# Out: ...

#<div class="col-md-4 wow fadeInUp">
# <div class="stellenagebot">
#  <h3>
#   Facharzt (m/w/d) für Augenheilkunde in Voll- oder Teilzeit
#  </h3>
#  <h4>
#   Erfurt
#  </h4>
#  <a class="btn btn-default" href="/Stellenangebot/Facharzt_Augenheilkunde_Erfurt/">
#   Mehr
#  </a>
# </div>
#</div>
#<div class="col-md-4 wow fadeInUp">
# <div class="stellenagebot">
#  <h3>
#   Facharzt (m/w/d) für Augenheilkunde in Voll- oder Teilzeit
#  </h3>
#  <h4>
#   Eschwege
#  </h4>
#  <a class="btn btn-default" href="/Stellenangebot/Facharzt_Augenheilkunde_Eschwege/">
#   Mehr
#  </a>
# </div>
#</div>

# ...

print(soup.find_all('.stellenagebot'))
# Out: []

print(soup.find_all(string=re.compile("Stellenangebot")))
# ['Stellenangebote Facharzt für Augenheilkunde und Karrierewege für Ärzte', '<h3>Zur Verst&auml;rkung unseres Teams suchen wir:</h3>\n\n<p class="hyphenate" lang="de">&nbsp;</p>\n\n<div id=\'jobauflistung\' class=\'row\'><div class=\'col-md-4 wow  fadeInUp\'><div class="stellenagebot">\n<h3>Facharzt (m/w/d) für Augenheilkunde in Voll- oder Teilzeit</h3>\n\n<h4>Aachen</h4>\n<a class="btn btn-default" href="/Stellenangebot/Facharzt_Augenheilkunde_Aachen/">Mehr</a></div></div><div class=\'col-md-4 wow  fadeInUp\'><div class="stellenagebot">\n<h3>Facharzt (m/w/d) fü
# ...
# ></div>\n</div>\n</div>\n</div>\n</footer>\n</div><!-- AUF ALLEN SEITEN IN DEN BODY-TAG EINFÜGEN ']

print(len(soup.find_all(string=re.compile("Stellenangebot"))))
# Out: 2

Also no luck with Selenium:

from selenium import webdriver

from job_scraper.configuration import CHROMEDRIVER_VERSION, ROOT_PATH, \
    CHROME_OPTIONS


driver = webdriver.Chrome(
    ROOT_PATH / f'assets/chrome_drivers/{CHROMEDRIVER_VERSION}',
    options=CHROME_OPTIONS
)
driver.maximize_window()
driver.get(url)
driver.find_elements_by_css_selector('.stellenagebot')
# Out: []