I have code that scrapes a website but does so in a way that after so many scrapes from a run, I get a 403 forbidden error. I understand there is a package in R called polite that does the work of figuring out how to run the scrape to the hosts requirements so the 403 won't occur. I tried my best at adapting it to my code but I'm stuck. Would really appreciate some help. Here is some sample reproducible code with just a few links from many:
library(tidyverse)
library(httr)
library(rvest)
library(curl)
urls = c("https://www.pro-football-reference.com/teams/pit/2021.htm", "https://www.pro-
football-reference.com/teams/pit/2020.htm", "https://www.pro-football-
reference.com/teams/pit/2019.htm")
pitt <- map_dfr(
.x = urls,
.f = function(x) {Sys.sleep(2); cat(1);
read_html(
curl(x, handle = curl::new_handle("useragent" = "chrome"))) %>%
html_nodes("table") %>%
html_table(header = TRUE) %>%
simplify() %>%
.[[2]] %>%
janitor::row_to_names(row_number = 1) %>%
janitor::clean_names(.) %>%
select(week, day, date, result = x_2, record = rec, opponent = opp, team_score = tm, opponent_score = opp_2) %>%
mutate(year = str_extract(string = x, pattern = "\\d{4}"))
}
)
This run should work no problem but the full run includes all years 1933-2021 instead of just the three year links provided in the example. I'm open to any way to responsibly scrape this using the polite package or any other an expert might be more familiar with.