Issue activating submit_form with checkboxes while webscraping in R

Viewed 92

I'm trying to scrape archival documents from a website that requires me to submit a search form with checkboxes in order to bring up a list of files. Here is an example link, note how the .pdfs only show up after clicking "Hae." When I submit the form, I want the results for all four check boxes to come up so that the .pdfs that contain "Liite" and "PTK" will also show up.

#Creates a dataset of URLs for download
url_root <- c("http://avoindata.eduskunta.fi/digitoidut/download/")

year_set <- c(1939:1951)

seq <- length(year_set)

#Script for downloading URLs, including years of interest
download_urls <- vector(mode = "character", length = length(year_set))
  for(i in 1:13) 
    {
  download_urls[[i]] <- paste(url_root, year_set[[i]], sep = "")
  }

#Scraping data

#Creates vector for input of download URLs
finnish_urls <- vector(mode="character")

for(i in 1:13) {
  url <- c(download_urls[i])
  test.session <- html_session(url) #Since these webpages require me to submit a "form" to click a button, I have to create a new HTML session and submit that filled form to the server
  form <- html_form(test.session)[[1]]
  form$fields[6]$type$checked <- TRUE #Sets "Asiakirjat" checkbox to true
  form$fields[3]$type$checked <- TRUE #Sets "Pöytäkirjat" checkbox to true
  form$fields[4]$type$checked <- TRUE #Sets "Liite" checkbox to true
  form$fields[5]$type$checked <- TRUE #Sets "Hakemisto" checkbox to true
  form$url <- url
  filled.form <- set_values(form)
  test.session$url <- url
  filled.form$url <- url

  test.session2 <- submit_form(test.session, filled.form, submit = filled.form$name)

  #Pulls data content from new submitted page
  updated.session <- html_nodes(test.session2, ".main-content a") #%>%
  html_attr("href") #Extracts links

  #Removes non-Finnish links (the PDFs are duplicated in Swedish)
  for(j in 1:length(updated.session)) {
  if(grepl(x = updated.session[[j]], pattern = "suomi")) {
    finnish_urls <- c(finnish_urls, updated.session[[j]])
  }
}
}

Currently, finnish_urls includes all the desired years, but only the results from "Asiakirjat" (i.e. urls with ASK in them are showing up). Any idea what I might be doing wrong?

This unresolved issue seems to be closest to what I'm dealing with.

Thanks for your help!

0 Answers
Related