WebDriverWait().until().send_keys giving TimeoutException

Anthony Huynh 0
Viewed 26

I am trying to scrape a website and I am unable to send keys to the email/password boxes to log in.

from config import root_dir, username_placer, password_placer

import os
import pathlib
os.chdir(root_dir)

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import Select 

import time
import math
import datetime as dt
import pandas as pd
from parse_account_name import parse_account_name


class Placer:

    def __init__(self, account_name):
        # Set webdriver parameters
        chrome_options = webdriver.ChromeOptions()
        chrome_options.add_argument('--kiosk-printing')
        chrome_options.add_argument("window-size=1440,1080")
        chromedriver = r'.\_reference\chromedriver.exe'
        self.driver = webdriver.Chrome(executable_path=chromedriver, options=chrome_options)
        self.driver.get('https://permits.placer.ca.gov/CitizenAccess/Default.aspx')       
        # Log in
        WebDriverWait(self.driver, 30).until(lambda d: d.find_element_by_xpath('//*[@id="ctl00_PlaceHolderMain_LoginBox_txtUserId"]')).send_keys(username_placer)
        self.driver.find_element_by_xpath('//*[@id="ctl00_PlaceHolderMain_LoginBox_txtPassword"]').send_keys(password_placer)
        self.driver.find_element_by_xpath('//*[@id="ctl00_PlaceHolderMain_LoginBox_btnLogin"]').click()
        time.sleep(2)

def search_placer(account_name):
    scraper = Placer(account_name)

#

if __name__ == '__main__':
    account_name = '''test'''
    result = search_placer(account_name)
    print(result)

It will just return a TimeoutException. I've tried using EC.element_to_be_clickable, but I get the same error. Without waiting and just trying to skip to sending keys, I (predictably) get NoSuchElementException.

If I print the page source, it returns the collapsed HTML.

It seems like the page and elements load by the time it looks for the login box element, so I'm not sure why it can't find it. What should I try at this point?

selenium-webdriver selenium python
1 Answers

The issue you are facing is that the elements you are trying to reach are inside of an IFRAME. Using Selenium, you have to first switch to the IFRAME before interacting with that part of the page. Once you are done with elements inside the IFRAME, make sure you switch back to the default content.

Additional feedback...

  1. If you are going to locate an element using the ID, always use By.ID instead of By.XPATH. It's much simpler syntax and easier to read.
  2. You should always use the EC class methods when possible rather than writing your own custom wait conditions.
  3. Using sleep is a bad practice and should be avoided whenever possible. Instead use WebDriverWait each time you need to wait for an element as you did earlier in your code.

Using these suggestions, the updated code would be something like

self.driver.get('https://permits.placer.ca.gov/CitizenAccess/Default.aspx')       
driver.switch_to.frame(driver.find_element(By.ID, "ACAFrame"))
# Log in
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "ctl00_PlaceHolderMain_LoginBox_txtUserId")).send_keys(username_placer)
self.driver.find_element(By.ID, "ctl00_PlaceHolderMain_LoginBox_txtPassword").send_keys(password_placer)
self.driver.find_element(By.ID, "ctl00_PlaceHolderMain_LoginBox_btnLogin").click()
driver.switch_to.default_content()

# don't use sleep... instead use a WebDriverWait
JeffC0
Related