Python-Selenium : How can I get access to this part of the website?

user_16201212 0
Viewed 74

I recently started learning web scraping. I wanted to have some information about the Job ads in the "Indeed" website. My problem is that when I click on the job name, the site shows the job description on the right side. I want to have access to this part to get information out but I run into this problem every time :

Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="jobDescriptionText"]"}

I tried other ways , but I couldn't solve it , so How can I get access to that part without getting that Error?

driver.get('https://www.indeed.com/')

def job_city():
    job_name = driver.find_element_by_css_selector("input[name='q']")
    job_name.clear()
    city = driver.find_element_by_css_selector("input[name='l']")
    city.clear()
    job_name.send_keys('Data scientist')
    city.send_keys('New york')
    driver.find_element_by_class_name('icl-WhatWhere-buttonWrapper').click()
    #time.sleep(10)

job_city()

main = driver.find_element_by_id('mosaic-provider-jobcards')
n_jobs = main.find_elements_by_tag_name('a')

jobs = list()
links = list()
#I wanted to have the job links so I made this loop
for num in range(len(n_jobs)):
    if (len(n_jobs[num].get_attribute('id')))==0:
        pass
    else:
        links.append(n_jobs[num].get_attribute('href'))
        jobs.append(n_jobs[num])

i = 0  
for job in jobs:
    title = job.find_element_by_tag_name('h2').find_elements_by_tag_name('span')
    job.find_element_by_tag_name('h2').click()
    time.sleep(3)
    info_right_side = driver.find_element_by_id('jobDescriptionText')
.
.
.
web-scraping selenium python
1 Answers

This id

jobDescriptionText

is in iframe.

in order to interact with elements which are part of iframe, driver has to switch it focus into it.

wait = WebDriverWait(driver, 10)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.ID, "vjs-container-iframe")))
info_right_side = driver.find_element_by_id('jobDescriptionText')

Imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Once, you are done with iframe, you'd need to switch to default content, in order to interact with other web elements which aren't part of iframe.

driver.switch_to.default_content()
user_94440730
Related