Confusion Regarding HTML Code For Web Scraping With R

user_10962789 0
Viewed 96

I am struggling using the rvest package in R, most likely due to my lack of knowledge about CSS or HTML. Here is an example (my guess is the ".quote-header-info" is what is wrong, also tried the ".Trsdu ..." but no luck either):

library(rvest)
url="https://finance.yahoo.com/quote/SPY"

website=read_html(url) %>%
  html_nodes(".quote-header-info") %>%
  html_text() %>% toString()

website

The below is the webpage I am trying to scrape. Specifically looking to grab the value "416.74". I took a peek at the documentation here (https://cran.r-project.org/web/packages/rvest/rvest.pdf) but think the issue is I don't understand the breakdown of the webpage I am looking at.

enter image description here

rvest web-scraping r
2 Answers

The tricky part is determining the correct set of attributes to only select this one html node.

In this case the span tag with a class of Trsdu(0.3s) and Fz(36px)

library(rvest)
url="https://finance.yahoo.com/quote/SPY"

#read page once
page <- read_html(url)

#now extract information from the page
price <- page %>%  html_nodes("span.Trsdu\\(0\\.3s\\).Fz\\(36px\\)") %>%
   html_text()

price

Note: "(", ")", and "." are all special characters thus the need to double escape "\\" them.

user_57922440

Those classes are dynamic and change much more frequently than other parts of the html. They should be avoided. You have at least two more robust options.

  1. Extract the javascript option housing that data (plus a lot more) in a script tag then parse with jsonlite
  2. Use positional matching against other, more stable, html elements

I show both below. The advantage of the first is that you can extract lots of other page data from the json object generated.

library(magrittr)
library(rvest)
library(stringr)
library(jsonlite)

page <- read_html('https://finance.yahoo.com/quote/SPY')

data <- page %>% 
  toString() %>% 
  stringr::str_match('root\\.App\\.main = (.*?[\\s\\S]+)(?=;[\\s\\S]+\\(th)') %>% .[2]

json <- jsonlite::parse_json(data)
print(json$context$dispatcher$stores$StreamDataStore$quoteData$SPY$regularMarketPrice$raw)
print(page %>% html_node('#quote-header-info div:nth-of-type(2) ~ div div:nth-child(1) span') %>% html_text() %>% as.numeric())
user_62412350
Related