How to scrape the login website using selenium

user_17050039 0
Viewed 261
from selenium import webdriver

from time import sleep

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.3"
}

driver= webdriver.Chrome('C:\Program Files (x86)\chromedriver.exe')
driver.get('https://www.counselingcalifornia.com/Find-a-Therapist')
driver.find_element_by_id("language_field").click("option",value="ENG")
driver.find_element_by_id(searchBtn).click()
print("successfully")

I am trying to scrape data but first Click on Language, select English, then click advanced search but they will give me error these is website https://www.counselingcalifornia.com/Find-a-Therapist

web-scraping selenium python
2 Answers

The reason is iframe.

language drop down is in iframe. In Selenium automation, if the webelements are wrapped inside an iframe, we should always switch to iframe first then we can interact with the elements.

Code :

driver = webdriver.Chrome(driver_path)
driver.maximize_window()

wait = WebDriverWait(driver, 30)

driver.get("https://www.counselingcalifornia.com/Find-a-Therapist")

wait.until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, "iframe[id$='IFrame_htmIFrame']")))
select = Select(wait.until(EC.visibility_of_element_located((By.ID, "language_field"))))
select.select_by_value('ENG')

wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a#searchBtn"))).click()

Imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.select import Select
user_94440730

If there was error related to the chromedriver version, then please install the latest version of the driver. You have a syntax error in your code. Include your searchBtn id in double-quotes.

driver.find_element_by_id("searchBtn").click()

from selenium import webdriver

from time import sleep

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.3"
}

driver= webdriver.Chrome('C:\Program Files (x86)\chromedriver.exe')
driver.get('https://www.counselingcalifornia.com/Find-a-Therapist')
driver.find_element_by_id("language_field").click("option",value="ENG")
driver.find_element_by_id("searchBtn").click()
print("successfully")
user_134097710
Related