Web Scraping XHR Dynamic pages with rvest and R

user_4871383 0
Viewed 1315

I am attempting to scrape a Dynamic website Morningstar.com via XHR requests.

The exact site I am scraping is: http://performance.morningstar.com/funds/etf/total-returns.action?t=SPY&region=USA&culture=en_US

What I am trying to scrape is the Quarterly performance number (1-month). The result should be 0.64 as of today. 

try(res <- GET(url = "http://performance.morningstar.com/fund/performance-return.action",
                 query = list(
                   t="SPY",
                   region="usa",
                   culture="en-US"
                 )
  ))

  tryCatch(x <- content(res) %>%
             html_nodes(xpath = '//*[@id="tab-quar-end-content"]/table/tbody/tr[1]/td[1]') %>% 
             html_text() %>%
             trimws() %>% 
             as.numeric()
           , error = function(e) x <-NA)

However, the result is numeric(0)

Any idea what I am doing wrong?

Sody

Update:

I was able to get the html data with the following code: 

 try(res <- GET(url = "http://performance.morningstar.com/fund/performance-return.action",
                 query = list(

                   t = "SPY",
                   region = "usa",
                   culture = "en-US",
                   ops = "clear",
                   s = "0P0000J533",
                   ndec = "2",
                   ep = "true",
                   align = "q",
                   annlz = "true",
                   comparisonRemove = "false"

                 )
  ))

But I am still having problems pointing to the data using either the CSS selector or the xpath with rvest.

What do you guys use to find those data points? is SelectorGadget still the go to?

Cheers, Aaron

rvest web-scraping r
2 Answers
library(httr)

GET(
  url = "http://performance.morningstar.com/perform/Performance/cef/trailing-total-returns.action", 
  add_headers(
    Referer = "http://performance.morningstar.com/funds/etf/total-returns.action?t=SPY&region=USA&culture=en_US", 
    `X-Requested-With` = "XMLHttpRequest"
  ),
  query = list(
    t = "ARCX:SPY", region = "usa", culture = "en-US", 
    cur = "", ops = "clear", s = "0P00001MK8", ndec = "2", ep = "true", 
    align = "q", annlz = "true", comparisonRemove = "false", 
    benchmarkSecId = "", benchmarktype = ""
  ),
  verbose()
) -> res

You have to target the XHR directly.

user_14570510

The table is embedded using java script, not hard-coded. You won't be able to scrape this data.

user_86294150
Related