Unable to click on next button with Selenium

Question

I've been at this for hours and haven't made any progress. I'm trying to click on the next button on this page here

Here's my code:

#!/usr/local/bin python3

import sys
import time
import re
import logging
from selenium import webdriver
from selenium.webdriver.firefox.options import Options as options
from bs4 import BeautifulSoup as bs
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.action_chains import ActionChains


_USE_VIRTUAL_DISPLAY = False
_FORMAT = '%(asctime)s - %(levelname)s - %(name)s - %(message)s'
# logging.basicConfig(filename=LOG_FILENAME,level=logging.DEBUG)
logging.basicConfig(format=_FORMAT, level=logging.INFO)
_LOGGER = logging.getLogger(sys.argv[0])
_DEFAULT_SLEEP = 0.5


try:
    options = options()
    # options.headless = True
    

    driver = webdriver.Firefox(options=options, executable_path=r"/usr/local/bin/geckodriver")
    
    print("Started Browser and Driver")


except:
    _LOGGER.info("Can not run headless mode.")

url = 'https://www.govinfo.gov/app/collection/uscourts/district/alsd/2021/%7B%22pageSize%22%3A%22100%22%2C%22offset%22%3A%220%22%7D'

driver.get(url)
time.sleep(5)

page = driver.page_source
soup = bs(page, "html.parser")


next_page = WebDriverWait(driver,5).until(EC.element_to_be_clickable((By.XPATH,'//*[@id="collapseOne1690"]/div/span[1]/div/ul/li[8]/a')))
if next_page:
    print('*****getting next page*****')
    # driver.execute_script('arguments[0].click()', next_page)
    next_page.click()
    time.sleep(3)
    
else:
    print('no next page')
    

driver.quit()

I get a timeout error. I've tried changing the XPath. I've tried ActionChains to scroll into view and none have worked. Any help appreciated.

user_12730112 · Answer

1 Your XPATH does not work because it uses dynamic class name collapseOne1690, as was mentioned earlier. Also, it's not very stable even if you used a part of this class name. If you prefer XPaths, I'd suggest this one: //span[@class='custom-paginator']//li[@class='next fw-pagination-btn']/a or just //li[@class='next fw-pagination-btn']/a. You can also use css selector: .next.fw-pagination-btn

2 I got rid of logging code because it also has some issues, re-check it.

3 5 seconds explicit wait is too small. Make it at least 10 seconds, better 15. It's just a suggestion.

The smallest reproducible code which clicks the button and uses Firefox is:

from selenium import webdriver
from selenium.webdriver.firefox.options import Options as options
from bs4 import BeautifulSoup as bs
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By

options = options()
# options.headless = True

driver = webdriver.Firefox(options=options)

print("Started Browser and Driver")

url = 'https://www.govinfo.gov/app/collection/uscourts/district/alsd/2021/%7B%22pageSize%22%3A%22100%22%2C%22offset%22%3A%220%22%7D'

driver.get(url)

page = driver.page_source
soup = bs(page, "html.parser")
print(soup)

next_page = WebDriverWait(driver, 15).until(
    EC.element_to_be_clickable((By.XPATH, "//span[@class='custom-paginator']//li[@class='next fw-pagination-btn']/a")))
next_page.click()

# driver.quit()

user_13770978 · Answer

It appears when I load up this page that the div id's are assigned dynamically. The first time I loaded the page, the id was collapseOne5168, the second time it was collapseOne1136

You might consider using find_element_by_class_name("next fw-pagination-btn") instead?

Unable to click on next button with Selenium

2 Answers