I am trying to scrape a javascript rendered table and after trying both selenium and phantomJS I've decided that JSON would be the easiest way to do it. However I am quite new to R and not very good at handling lists, and because of that I cannot get my data into the table format I desire. I've looked at a number of solutions but for some reason they don't really work on the JSON I have.
The JSON data is rendered through this URL. And this is the actual website where the table is located.
What I've done so far is to try to parse the JSON into R and coerce it into a dataframe, based on what I've seen from most answers on stackoverflow.
library(httr)
library(jsonlite)
rf <- GET(url) #the entire URL is very long so I'm not putting it here
rfc <- content(rf)
Doing this returns me a large list of four elements, rfc. I then apply the following function.
library(httr)
library(jsonlite)
json_file <- lapply(rfc, function(x) {
x[sapply(x, is.null)] <- NA
unlist(x)
})
This returns me an error:
Error in x[sapply(x, is.null)] <- NA : invalid subscript type 'list'
Given that I only need the second element of the list, which is where the information is at, I attempt to subset it:
json_file <- lapply(rfc[2], function(x) {
x[sapply(x, is.null)] <- NA
unlist(x)
})
This returns me a large list, 12mb in size. When I try to coerce it to a dataframe using as.data.frame, R returns me 506472 observations of 1 variable. The different columns have all been squashed into one and the headers are gone.
Can anyone tell me how I should go about doing this? There's a free online JSON to CSV converter here that does exactly what I need beautifully. This is what it produces:
Unfortunately this is not a solution. Because I intend to run this in Shiny I want to do everything within R. Any help is appreciated, thanks.