How do I get last page using Rvest if not identified on page?

An interesting problem. I decided to see if there was an API. There are a number of APIs. See: https://openparliament.ca/api/

Not spending much time reading, I opted for the bills API endpoint. I couldn't easily see a request that returned the total bills count, or all bills in one go. I did find I could use one of the endpoints in a loop, using a querystring construct to return 500 (max I could get to return in one) bills at a time, using offset to get the next batch. I used a for loop (gasp), as I was unsure how else to terminate making requests when next_url, in the response json, was NULL.

library(jsonlite)
library(data.table)
library(dplyr)

n <- 0
result <- list()

while (TRUE) {
  url <- sprintf("https://api.openparliament.ca/bills/?limit=500&offset=%i&format=json", n)
  data <- jsonlite::read_json(url, simplifyVector = T)
  next_url <- data$pagination$next_url
  df <- data$objects %>% mutate(name_en = name$en, name_fr = name$fr) %>% select(-c('name'))
  result[[(n/500)+1]] <- df
  if (is.null(next_url)) {
    break
  } else {
    n <- n + 500
  }
}

df <- rbindlist(result)
df %>% arrange(desc(introduced)) %>% head()

df contains the data you see on the page for all bills listed via API. Currently 6,620 rows.

Py:

import requests
import pandas as pd

n = 0
results = []

def get_dict_list(value: list)->list:
    output = []
    
    for i in value:
        d = {k:v for k,v in i.items() if k != 'name'}
        d['name_en'] = i['name']['en']
        d['name_fr'] = i['name']['fr']
        output.append(d)
    return output
    
with requests.Session() as s:
    while True:
        url = f'https://api.openparliament.ca/bills/?limit=500&offset={n}&%5D=&format=json'
        r = s.get(url)
        data = r.json()
        next_url = data['pagination']['next_url']
        results.extend(get_dict_list(data['objects']))
        
        if next_url is None:
            break

        n+=500   
            
df = pd.DataFrame(results) 

I decided to try using RSelenium to navigate to the last page, and grab the last page number. This did the trick, but was not very elegant or efficient. I allow myself this as someone who is new to R and webscraping.

#Step 1 - call packages


library(tidyverse)
library(rvest)
library(RSelenium)

#Step 2 - Identify the main webpage

url <- 'https://www.parl.ca/legisinfo/Home.aspx'

#Step 3 - Open browser

rD <- rsDriver(port = 4546L, version = "latest", browser=c("firefox"))
remDr <- rD[["client"]]

#Step 4 - Navigate to page
remDr$navigate(url)

#Step 5 - Select filters

webElem <- remDr$findElement(using = 'css selector',".filterTopRemoveImage")
webElem$clickElement() #Click to remove default filter so that all Parliamentary sessions are included

webElem <- remDr$findElement(using = 'css selector',".filterRefiner~ div:nth-child(5) .filterRefinement")
webElem$clickElement() #Click to select only House of Commons Bills

#Step 6 - Get last page by navigating >>

webElem <- remDr$findElement(using = 'link text',">>")
webElem$clickElement()

html <- remDr$getPageSource()[[1]] #Get everything on the page
html <-read_html(html) #Parse HTML

lastpage <- html %>% html_nodes(".resultPageCurrent:nth-child(12)") %>% html_text()
lastpage