R webscraping doesn't work when url is in variable

Viewed 58

I'm having trouble scraping in R. I want to scrape genre information for several titles on goodreads. If I do this, it works completely fine and gives me what I need:

library(polite)
library(rvest)
library(dplyr)

session <- bow("https://www.goodreads.com/book/show/29991718-royally-matched",
           delay = 5)
genres <- scrape(session) %>%
  html_elements(".bookPageGenreLink") %>% 
  html_text()
genres

However, since I'd like loop over several pages, I need this to work, but it always returns character(0).

host <- "https://www.goodreads.com/book/show/29991718-royally-matched"
session <- bow(host,
               delay = 5)
genres <- scrape(session) %>%
  html_elements(".bookPageGenreLink") %>% 
  html_text()
genres

Something like this would also be fine for me, but it doesn't work either:

link = "29991718-royally-matched"
session <- bow(paste0("https://www.goodreads.com/book/show/29991718-royally-matched", link),
               delay = 5)
genres <- scrape(session) %>%
  html_elements(".bookPageGenreLink") %>% 
  html_text()
genres

If I open the website and disable javascript, it still works completely fine, so I don't think Selenium is necessary and I really can't figure out why this doesn't work, which drives me crazy.

Thank you so much for your support!

Solution (kind of)

So I noticed that the success of my scrapings was kind of dependent on the random moods of the scraping gods. So I did the following:

links <- c("31752345-black-mad-wheel", "00045101-The-Mad-Ship", "2767052-the-hunger-games", "18619684-the-time-traveler-s-wife", "29991718-royally-matched")
data <- data.frame(links)
for (link in links) {
  print(link)
  genres <- character(0)
  url <- paste0('https://www.goodreads.com/book/show/',link)
#I don't know why, but resaving it kinda helped
  host <- url
#I had the theory that repeating the scraping would eventually lead to a result. For me that didn't work though
  try <- 0 
  while (identical(genres, character(0)) & (try < 10)) {
    try <- try+1
    print(paste0(try, ": ", link))
    session <- bow(host,
                   delay = 5)
    scraping <- scrape(session)
    genres <-  scraping %>%  html_elements(".bookPageGenreLink") %>% 
      html_text()
  }
    if(identical(genres, character(0))){
    print("Scraping unsuccessfull.. :( ")
  }
  else{
    print("scraping success!!")
    genres.df <- data.frame(genres)
    data <- left_join(data, 
    genres.df, by = c("link"))
  }
    }
## then I created a list of the missing titles
missing_titles <- data %>% 
  filter(is.na(genre_1)) 
missing_links <- unique(missing_titles$link)

So the next step(s) were closing R (while saving the workspace of course), restarting it and refeeding the loop with missing_titles instead of links. It took me like 7 iterations of that to get everything I needed, while on the last run I had to insert the last remaining link directly into example 1, since it did not work inside the loop. whyever. I hope the code kind of works, since I wanted to spare you pages of wild data formatting.

If someone has an explanation, why I needed to go through this hustle, I would still very much appreciate it.

1 Answers

You can consider to use the R package RSelenium as follows :

library(RSelenium)
library(rvest)
url <- "https://www.goodreads.com/book/show/29991718-royally-matched"
shell('docker run -d -p 4445:4444 selenium/standalone-firefox')
remDr <- remoteDriver(remoteServerAddr = "localhost", port = 4445L, browserName = "firefox")
remDr$open()
remDr$navigate(url)
page_Content <- remDr$getPageSource()[[1]]
read_html(page_Content) %>% html_elements(".bookPageGenreLink") %>% html_text()

Afterwards, you can loop over the url you want.

Related