Struggling to Scrape Table using rvest Package

user_6636902 0
Viewed 47

I've recently starting using R again after a long hiatus and I'm extremely rusty, especially when it comes to html and scraping data (w/rvest).

My main issue right now is identifying the correct nodes/'XPath' to input into my function to get it to pull the correct data, more specifically, I'm trying to scrape college hockey play-by-play data from the NCAA website (example)....

I've tried numerous approaches without any success.. for example:

Since it seems each period's data is nested in separate "div.play-by-play-period" classes(?), I tried focusing on scraping one period, then building from there...

So I drilled down to the table containing the 1st period's data ('tbody') and copy the xpath and pasted into the code below:

url <- "https://www.ncaa.com/game/5935492/play-by-play"

gm <- url %>%
  read_html() %>%
  html_nodes(xpath='//*[@id="gamecenterAppContent"]/div/div[3]/div[2]/table/tbody') %>%
  html_table()

Resulting in a "List of 0"...

Any help would be greatly appreciated!

rvest web-scraping r html
1 Answers

Easier for you to call on their source with httr2.

library(tidyverse)
library(httr2)

data <- "https://data.ncaa.com/casablanca/game/5935492/pbp.json" %>%  
  request() %>%  
  req_perform() %>%  
  resp_body_json()

Or with jsonlite::fromJSON

jsonlite::fromJSON("https://data.ncaa.com/casablanca/game/5935492/pbp.json")

Parse it

data <- jsonlite::fromJSON("https://data.ncaa.com/casablanca/game/5935492/pbp.json") %>% 
  .$periods %>% 
  .$playStats %>% 
  bind_rows() %>% 
  as_tibble()

# A tibble: 111 x 4
   score time  visitorText                                                  homeText      
   <chr> <chr> <chr>                                                        <chr>         
 1 ""    20:00 "Jack Watson at goalie for RPI."                             ""            
 2 ""    20:00 ""                                                           "Clay Stevens~
 3 ""    19:39 ""                                                           "Shot by DAR ~
 4 ""    19:36 "Penalty on Johnson, Jake RPI 2 minutes for CrossChecking."  ""            
 5 ""    18:04 ""                                                           "Shot by DAR ~
 6 ""    17:52 ""                                                           "Shot by DAR ~
 7 ""    16:25 ""                                                           "Shot by DAR ~
 8 ""    16:25 "Penalty on Dubinsky, Zach RPI 2 minutes for CrossChecking." ""            
 9 ""    15:55 ""                                                           "Shot by DAR ~
10 ""    14:24 ""                                                           "Shot by DAR ~
# ... with 101 more rows
user_172133550
Related