I am trying to scrape PDFs from a newspaper archive that uses password protection, using rvest. The problem is that while I can log in in a session, I couldn't find a way to use the download.file() command within the session. The website is somewhat strange, so the only option I've found so far to actually log in was by actually using the PDF link.
Here is my current code (unfortunately, as I can't give out the login details, it is only limitedly reproducible):
#login to DLG archive
login <- "https://dlgarchiv.lv.de/pdf/616694ca9b67fDLG_14-19_10_2021.pdf"
pgsession <- session(login)
pgform <- html_form(pgsession)[[1]]
filled_form <- html_form_set(pgform, customerno="*******", plz="*****")
session_submit(filled_form)
#download file
download.file(url="https://dlgarchiv.lv.de/pdf/616694ca9b67fDLG_14-19_10_2021.pdf",destfile="test.pdf",mode="wb")
As a result, I am only downloading an HTML document of the log-in page, i.e. my log-in is being "forgotten" in between.
Looking at the output following the session_submit() command, I am actually getting to the PDF, but I can't download it...
<session> https://dlgarchiv.lv.de/pdf/616694ca9b67fDLG_14-19_10_2021.pdf
Status: 200
Type: application/pdf
Size: 1394950