R code for login to online news paper while scraping urls?

Viewed 34

I have a list of URLs, and I am trying to scrape the content from them for my research in R. I scrape all content using read_html in a for loop. The problem is that I need to login to the newspaper to scrape the content. So I am trying to login with my ID and password so that I can scrape the news content and date for each URL I found in the search result.

Can I somehow write login information into the for loop in order to access the content of the news articles?

library(rvest)
library(stringr)
library(purrr)
library(readbulk)
library(dplyr)

#Read URLs
urls <- read_bulk("C:/Users/XXXX", extension = ".csv") %>%
   dplyr::distinct(link) #removing dublicates

#For Loop
titles <- c()
text <- c()
url <- c()
date <- c()

for(i in 1:nrow(urls)){ 
  data <- read_html(paste0(urls$link[i]))
  body <- data %>%
    html_nodes("p") %>%
    html_text() %>%
    str_c(collapse = " ", sep = "")
  text = append(text, body)

  data <- read_html(paste0(urls$link[i]))
  header <- data %>%
    html_node("title") %>%
    html_text()
  titles = append(titles, rep(header,each=length(body)))

  data <- read_html(paste0(urls$link[i]))
  time <- data %>%
    html_nodes("time") %>% #See HTML source code for data within this tag
    html_text() %>%
    str_c(collapse = " ", sep = "")
  date = append(date, rep(time,each=length(time)))

  url = append(url, rep(paste0(urls$link[i]),each=length(body)))

  print(i)
}


data <- data.frame(Headline=titles, Body=text, Date=date, Url=url) # As Dataframe
0 Answers
Related