"Error in open.connection(x, "rb") : HTTP error 403" When trying to webscrape in R (2022)

College Kid 0
Viewed 31

I've looked at all the other answers to this error from other posts and none of them have solved my issue.

I'm trying to read the html data from this website in various ways:

curl::curl("https://www.usnews.com/best-colleges/rankings/national-universities") %>% read_html() 
url <- 'https://www.usnews.com/best-colleges/rankings/national-universities'
url('https://www.usnews.com/best-colleges/rankings/national-universities', "rb")

And they all give me the forbidden error. How can I work around this?

web-scraping r html
2 Answers

Here is an approach that can be considered :

library(RSelenium)
library(rvest)
url <- "https://www.usnews.com/best-colleges/rankings/national-universities"
shell('docker run -d -p 4445:4444 selenium/standalone-firefox')
remDr <- remoteDriver(remoteServerAddr = "localhost", port = 4445L, browserName = "firefox")
remDr$open()
remDr$navigate(url)
page_Content <- remDr$getPageSource()[[1]]
html <- read_html(page_Content)
Emmanuel Hamel0

Here is an approach that can be considered :

library(RDCOMClient)
url <- "https://www.usnews.com/best-colleges/rankings/national-universities"
IEApp <- COMCreate("InternetExplorer.Application")
IEApp[['Visible']] <- TRUE
IEApp$Navigate(url)
Sys.sleep(5)
doc <- IEApp$Document()
page_Content <- doc$documentElement()$innerHtml()
html <- read_html(page_Content)
Emmanuel Hamel0
Related