rvest how to get last page number in r language

user_14782971 0
Viewed 105

I'm learning web scraping and want to create an example for myself.

https://www.goodreads.com/search?page=1&qid=ckDrIeoJ2c&query=harry+potter&tab=books&utf8=%E2%9C%93

I want to scrape last page number which is 100 by using above url. I tried several different codes, but they are not working well.

url %>%
  read_html(x) %>%
  html_nodes('div.leftContainer') %>%
  html_nodes('a[href^="/search?page=100&qid=ckDrIeoJ2c&query=harry+potter&tab=books&utf8=%E2%9C%93"]') %>%
  html_text()

I used html_nodes to get text '100' but it failed. I want to use length() and as.integer() to get the number. I would like to know how to get the value of last page number.

rvest r html
2 Answers

You should be able to use nth-last-of-type to get penultimate href containing page

library(rvest)
url <- 'https://www.goodreads.com/search?page=1&qid=ckDrIeoJ2c&query=harry+potter&tab=books&utf8=%E2%9C%93'
last_page <- read_html(url) %>% html_node('[href*=page]:nth-last-child(2)') %>% html_text() %>% as.integer()
user_62412350

Below another possible solution:

library(RSelenium)
remDr <- rsDriver(port=4555L,browser = "firefox")
remoteDriver<- remDr[["client"]]
url <-  "https://www.goodreads.com/search?page=1&qid=ckDrIeoJ2c&query=harry+potter&tab=books&utf8=%E2%9C%93"
remoteDriver$navigate(url)

#gets the last number of page
last_page<-remoteDriver$findElement(using =  'xpath', value = '/html/body/div[2]/div[3]/div[1]/div[2]/div[2]/div[3]/div/a[10]')$getElementText()

print(last_page)
[[1]]
[1] "100"
user_99930280
Related