Python: Selenium Pagination only saves page 1 to list, but not page 2, 3, etc

user_14133845 0
Viewed 80

The selenium code runs and paginates through each page correctly (in the selenium pop up window you can see it go from page 1-356).

But it the "driver.page_source" only saves the table from the first page or first few pages. Any idea why just the first few pages get appended to the list rather than 1-356? Thank you very much!

UPDATE: It seems to work if I keep the selenium pop-up as the active window, but is there a way around this if I want to run multiple selenium windows at once?

import pandas as pd
from selenium import webdriver
driver = webdriver.Chrome()

url = 'https://www.assessedvalues2.com/SearchPage.aspx?jurcode=36'

page = driver.get(url)

time.sleep(3)

driver.find_element_by_xpath('//*[@id="ctl00_MainContent_BtnSearch"]').click()

time.sleep(2)

df_master = []

for i in range(1,357):
    driver.find_element_by_xpath('//*[@id="ctl00_MainContent_TxtPage"]').clear()
    driver.find_element_by_xpath('//*[@id="ctl00_MainContent_TxtPage"]').send_keys(str(i), Keys.ENTER)
    time.sleep(3)
    df = pd.read_html(driver.page_source)
    df = pd.concat(df)
    df_master.append(df)
    time.sleep(2)

    
df_master = pd.concat(df_master)
Full_Table_df = pd.DataFrame(df_master)
display(Full_Table_df)
selenium-webdriver selenium python-3.x pagination python
1 Answers

selenium is a bit hissy with send keys and the browser window not in foreground.

you can try sending a click before send_keys

elem = driver.find_element_by_xpath('//*[@id="ctl00_MainContent_TxtPage"]')
elem.clear()
elem.click()
elem.send_keys(str(i), Keys.ENTER)

but since you want to run in background anyway you should use headless driver

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.headless = True
driver = webdriver.Chrome(options=chrome_options)
user_3608990
Related