I'm trying to scrape archival documents from a website that requires me to submit a search form with checkboxes in order to bring up a list of files. Here is an example link, note how the .pdfs only show up after clicking "Hae." When I submit the form, I want the results for all four check boxes to come up so that the .pdfs that contain "Liite" and "PTK" will also show up.
#Creates a dataset of URLs for download
url_root <- c("http://avoindata.eduskunta.fi/digitoidut/download/")
year_set <- c(1939:1951)
seq <- length(year_set)
#Script for downloading URLs, including years of interest
download_urls <- vector(mode = "character", length = length(year_set))
for(i in 1:13)
{
download_urls[[i]] <- paste(url_root, year_set[[i]], sep = "")
}
#Scraping data
#Creates vector for input of download URLs
finnish_urls <- vector(mode="character")
for(i in 1:13) {
url <- c(download_urls[i])
test.session <- html_session(url) #Since these webpages require me to submit a "form" to click a button, I have to create a new HTML session and submit that filled form to the server
form <- html_form(test.session)[[1]]
form$fields[6]$type$checked <- TRUE #Sets "Asiakirjat" checkbox to true
form$fields[3]$type$checked <- TRUE #Sets "Pöytäkirjat" checkbox to true
form$fields[4]$type$checked <- TRUE #Sets "Liite" checkbox to true
form$fields[5]$type$checked <- TRUE #Sets "Hakemisto" checkbox to true
form$url <- url
filled.form <- set_values(form)
test.session$url <- url
filled.form$url <- url
test.session2 <- submit_form(test.session, filled.form, submit = filled.form$name)
#Pulls data content from new submitted page
updated.session <- html_nodes(test.session2, ".main-content a") #%>%
html_attr("href") #Extracts links
#Removes non-Finnish links (the PDFs are duplicated in Swedish)
for(j in 1:length(updated.session)) {
if(grepl(x = updated.session[[j]], pattern = "suomi")) {
finnish_urls <- c(finnish_urls, updated.session[[j]])
}
}
}
Currently, finnish_urls includes all the desired years, but only the results from "Asiakirjat" (i.e. urls with ASK in them are showing up). Any idea what I might be doing wrong?
This unresolved issue seems to be closest to what I'm dealing with.
Thanks for your help!