R: function to scrape multiple links on wayback machine not scraping all data

Viewed 26

I am trying to scrape multiple links using a function, but each time I run the code, it only returns the column headers and not the data that goes with it. The function worked just fine on the actual website but is giving me loads of trouble on the wayback machine. Is there anything I can adjust to correct this error?

library(rvest); library(tidyverse)
library(httr2); library(janitor); library(stringr)

link_text = "https://web.archive.org/web/timemap/json?url=https://www.bjjcompsystem.com/tournaments/1869/categories&matchType=prefix&collapse=urlkey&output=json&fl=original,mimetype,timestamp,endtimestamp,groupcount,uniqcount&filter=!statuscode:[45]..&limit=10000&_=1663136483842" %>% 
  request() %>% 
  req_perform() %>% 
  resp_body_json(simplifyVector = TRUE) %>% 
  as_tibble() %>% 
  row_to_names(1)
prefix = 'https://web.archive.org/web/20220000000000*/'
links = str_c(prefix, link_text$original)

tree_page <- function(url) {
  html <- read_html(url)
  data.frame(division = html %>% 
               html_nodes('.category-title__age-division') %>%
               html_text(trim = TRUE),
             gender = html %>% 
               html_nodes('.category-title__age-division+ .category-title__label') %>% 
               html_text(trim = TRUE),
             belt = html %>% 
               html_nodes('.category-title__label:nth-child(3)') %>%
               html_text(trim = TRUE),
             weight = html %>% 
               html_nodes('.category-title__label:nth-child(4)') %>%
               html_text(trim = TRUE),
             winner = html %>% 
               html_nodes('.match-card__competitor--red') %>%
               html_text(trim = TRUE),
             opponent = html %>% 
               html_nodes('hr+ .match-card__competitor') %>% 
               html_text(trim = TRUE)
             )
}

master.tree <- map_df(links[1:10], tree_page)
0 Answers
Related