Scraping web page that is only accesible after submitting aspx form

user_1086777 0
Viewed 216

I need to scrape a table that is viewable only after an aspx form is submitted: https://nces.ed.gov/ipeds/datacenter/DataFiles.aspx (with "All Years" and "All Surveys") selected. I've attempted to get the form using rvest, but it does not appear that it's grabbing the one I need:

require(rvest)
#> Loading required package: rvest
#> Loading required package: xml2

url <- "https://nces.ed.gov/ipeds/datacenter/DataFiles.aspx"

sesh <- html_session(url)

forms <- sesh %>% html_nodes("form") %>% html_form()

forms
#> [[1]]
#> <form> 'HeaderSearch' (GET /search/search_redirect.asp)
#>   <input text> 'Search': Search
#>   <input hidden> 'website': NCES
#>   <input submit> '': Go
#> 
#> [[2]]
#> <form> 'search-box' (GET http://nces.ed.gov/search)
#>   <input hidden> 'output': xml_no_dtd
#>   <input hidden> 'client': nces
#>   <input hidden> 'site': nces
#>   <input hidden> 'sitesearch': nces.ed.gov/ipeds
#>   <input text> 'q': Search IPEDS
#>   <input image> '':

Created on 2020-03-23 by the reprex package (v0.3.0)

The first list item is the header search bar. The second might be the form, but if so it doesn't have a submit value.

I could use some help either figuring out how to emulate that form submission so I can grab the table of files, or alternatively figuring out if there is a url that gets to the same results page.

rvest r
1 Answers

This is tricky, but possible.

The first difficulty you have is that when you send a GET request (via html_session) to the url "https://nces.ed.gov/ipeds/datacenter/DataFiles.aspx", you are sending it without any session cookies. This makes the server redirect you to a different page, "https://nces.ed.gov/ipeds/use-the-data", and it is this page that you are seeing in your variable sesh.

However, since rvest (actually httr underneath rvest) re-uses session handles, all you need to do to overcome this problem is navigate to the login page, which allows httr to pick up the session cookies you need to browse as an anonymous user.

Here, we will also set our user agent to firefox.

library(httr)
library(rvest)
library(tibble)

url1    <- "https://nces.ed.gov/ipeds/datacenter/login.aspx?gotoReportId=8"
url2    <- "https://nces.ed.gov/ipeds/datacenter/DataFiles.aspx"

UA      <- "Mozilla/5.0 (Windows NT 6.1; rv:75.0) Gecko/20100101 Firefox/75.0"

html <- GET(url1, user_agent(UA))
html <- GET(url2, user_agent(UA))
page <- html %>% read_html()

Now page contains the page with the form that you want to submit. And this is where we come to the second difficulty. The easiest way to send a form is with rvest::submit_form(), but that doesn't seem to work because not all the fields are complete. We therefore need to build the form manually using rvest's scraping tools:

form <- list(`__VIEWSTATE` = page %>%
                html_node(xpath = "//input[@name='__VIEWSTATE']") %>%
                html_attr("value"),
             `__VIEWSTATEGENERATOR` = page %>%
                html_node(xpath = "//input[@name='__VIEWSTATEGENERATOR']") %>%
                html_attr("value"),
             `__EVENTVALIDATION` = page %>%
                html_node(xpath = "//input[@name='__EVENTVALIDATION']") %>%
                html_attr("value"),
             `ctl00$contentPlaceHolder$ddlYears` = "-1",
             `ddlSurveys` = "-1",
             `ctl00$contentPlaceHolder$ibtnContinue.x` = sample(50, 1),
             `ctl00$contentPlaceHolder$ibtnContinue.y` = sample(20, 1))

We can now submit this form, but before we do so, we need to add some headers, without which the server will throw a http 500:

Headers <- add_headers(`Accept-Encoding` = "gzip, deflate, br", 
                       `Accept-Language` = "en-GB,en;q=0.5", 
                       `Connection` = "keep-alive", 
                       `Host` = "nces.ed.gov", 
                       `Origin` = "https://nces.ed.gov", 
                       `Referer` = url2, 
                       `Upgrade-Insecure-Requests` = "1")

Finally, there is a cookie that is normally added via javascript that we will need to add manually:

Cookies <- set_cookies(setNames(c(cookies(html)$value, "true"),
                                c(cookies(html)$name, "fromIpeds")))

Now we can post our form with the correct form, headers and cookies to get the page you wanted:

Result  <- POST(url2, body = form, user_agent(UA), Headers, Cookies)

You can now scrape this page however you like. As an example, I will show that the text of the results table can be scraped quite easily:

Result %>% 
 read_html() %>% 
 html_node("#contentPlaceHolder_tblResult") %>% 
 html_table() %>%
 as_tibble()
#> # A tibble: 1,090 x 7
#>     Year Survey    Title        `Data File` `Stata Data Fil~ Programs Dictionary
#>    <int> <chr>     <chr>        <chr>       <chr>            <chr>    <chr>     
#>  1  2018 Institut~ Directory i~ HD2018      HD2018_STATA     SPSS, S~ Dictionary
#>  2  2018 Institut~ Educational~ IC2018      IC2018_STATA     SPSS, S~ Dictionary
#>  3  2018 Institut~ Student cha~ IC2018_AY   IC2018_AY_STATA  SPSS, S~ Dictionary
#>  4  2018 Institut~ Student cha~ IC2018_PY   IC2018_PY_STATA  SPSS, S~ Dictionary
#>  5  2018 Institut~ Response st~ FLAGS2018   FLAGS2018_STATA  SPSS, S~ Dictionary
#>  6  2018 12-Month~ 12-month un~ EFFY2018    EFFY2018_STATA   SPSS, S~ Dictionary
#>  7  2018 12-Month~ 12-month in~ EFIA2018    EFIA2018_STATA   SPSS, S~ Dictionary
#>  8  2018 12-Month~ Response st~ FLAGS2018   FLAGS2018_STATA  SPSS, S~ Dictionary
#>  9  2018 Admissio~ Admission c~ ADM2018     ADM2018_STATA    SPSS, S~ Dictionary
#> 10  2018 Admissio~ Response st~ FLAGS2018   FLAGS2018_STATA  SPSS, S~ Dictionary
#> # ... with 1,080 more rows

Created on 2020-03-31 by the reprex package (v0.3.0)

user_125003150
Related