Scraper doesn't stop clicking on the next page button

Viewed 140

I've written a script in python in combination with selenium to get some names and corresponding addresses displayed upon a search and the search keyword is "Saskatoon". However, the data, in this case, traverse multiple pages. My script almost does everything except for one thing.

  1. It still runs even though there are no more pages to traverse. The last page also holds ">" sign for next page option and is not grayed out.

Here is the link: Page_link

Search_keyword: Saskatoon (in the city/town field).

Here is what I've written:

from selenium import webdriver; import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
driver.get("above_link")
time.sleep(3)

search_input = driver.find_element_by_id("cityField")
search_input.clear()
search_input.send_keys("Saskatoon")
search_input.send_keys(Keys.ENTER)

while True:
    try:
        wait.until(EC.visibility_of_element_located((By.LINK_TEXT, "›"))).click()
        time.sleep(2)
    except:
        break
driver.quit()

BTW, I've just taken out the name and address part form this script which I suppose is not relevant here. Thanks.

1 Answers

You can use class attribute of > button as on last page it is "ng-scope disabled" while on rest pages - "ng-scope":

wait.until(EC.visibility_of_element_located((By.XPATH, "//li[@class='ng-scope']/a[.='›']"))).click()
Related