moving to the next page till the end with selenium in python

user_16832397 0
Viewed 38

I want to retrieve from the link below the first page can be retrieve but I have a problem for putting the loop for the next page till the end. May you help me and complete my code?

My link is: https://www150.statcan.gc.ca/n1/pub/71-607-x/2021004/exp-eng.htm?r1=(1)&r2=0&r3=0&r4=12&r5=0&r7=0&r8=2022-02-01&r9=2022-02-01

from selenium import webdriver
import time

url = "https://www150.statcan.gc.ca/n1/pub/71-607-x/2021004/exp-eng.htm?r1=(1)&r2=0&r3=0&r4=12&r5=0&r7=0&r8=2022-02-01&r9=2022-02-01"

driver = webdriver.Chrome("C:\Program Files\Python310\chromedriver.exe")
driver.get(url)
table = driver.find_element_by_id('report_table')
body = table.find_element_by_tag_name('tbody')
cells = body.find_elements_by_tag_name('td')

for cell in cells:
    print(cell.text)

it brings the first page data but I don't know how to retrieve the others.

web-scraping selenium python
1 Answers

Look for the next-page selector and iterate over it, if it is there, let it click after your extraction part. You wanna do that in a while loop for example which you break if selector can't be found.

from selenium import webdriver
import time

url = "https://www150.statcan.gc.ca/n1/pub/71-607-x/2021004/exp-eng.htm?r1=(1)&r2=0&r3=0&r4=12&r5=0&r7=0&r8=2022-02-01&r9=2022-02-01"

driver = webdriver.Chrome("C:\Program Files\Python310\chromedriver.exe")
driver.get(url)
table = driver.find_element_by_id('report_table')
body = table.find_element_by_tag_name('tbody')
cells = body.find_elements_by_tag_name('td')

for cell in cells:
    print(cell.text)
while True:
next_page = driver.find_element(By.XPATH, '//a[@id="report_results_next"]')
if next_page:
   # steps to extract if next_page 
   driver.get(next_page)
   table = driver.find_element_by_id('report_table')
   body = table.find_element_by_tag_name('tbody')
   cells = body.find_elements_by_tag_name('td')

   for cell in cells:
      print(cell.text)
else:
   # stop 
   break

This is not tested.

user_146273520
Related