Web scraping with R (rvest)

You can use :contains to target the appropriate table by class and then a substring that the table contains. Furthermore, you can use html_table() to extract in tabular format from matched node. You can then subset on a vector of desired columns. I don't know the correct football terms so have guessed the columns to subset on. You can adjusted the columns vector.

If you wrap the years and constructed urls to make requests to inside of a map2_dfr() call you can return a single DataFrame for all desired years.

library(tidyverse)
library(rvest)

years <- 2003:2021
urls <- paste("https://pt.wikipedia.org/wiki/Campeonato_Brasileiro_de_Futebol_de_", years, sep = "")
columns <- c("Pos.", "Equipes", "GP", "GC", "SG")

df <- purrr::map2_dfr(urls, years, ~
read_html(.x, encoding = "utf-8") %>%
  html_element('.wikitable:contains("ou rebaixamento")') %>%
  html_table() %>%
  .[columns] %>%
  mutate(year = .y, SG = as.character(SG)))

You can get all the tables from those links by doing this:

tabela <- function(link){
  read_html(link) %>% html_nodes("table.wikitable") %>% html_table()
}

all_tables = lapply(links_temporadas, tabela)
names(all_tables)<-2003:2022

This gives you a list of length 20, named 2003 to 2022 (i.e. one element for each of those years). Each element is itself a list of tables (i.e. the tables that are available at that link of links_temporadas. Note that the number of tables avaialable at each link varies.

lengths(all_tables)
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 
   6    5   10    9   10   12   11   10   12   11   13   14   17   16   16   16   16   15   17    7 

You will need to determine which table(s) you are interested in from each of these years.

Here is a way. It's more complicated than your function because those pages have more than one table so the function returns only the tables with a column names matching "Pos.".

Then, before rbinding the tables, keep only the common columns since the older tables have one less column, column "M".

suppressPackageStartupMessages({
  library("dplyr")
  library("rvest")
})

link_wikipedia <- "https://pt.wikipedia.org/wiki/Campeonato_Brasileiro_de_Futebol_de_2002"
pagina_wikipedia <- read_html(link_wikipedia)

links_temporadas <- pagina_wikipedia %>%
  html_nodes("tr:nth-child(9) div a") %>%
  html_attr("href") %>%
  paste("https://pt.wikipedia.org", ., sep = "")


tabela <- function(link){
  pagina_tabela <- read_html(link)

  lista_wiki <- pagina_tabela %>%
    html_elements("table.wikitable") %>%
    html_table()

  i <- sapply(lista_wiki, \(x) "Pos." %in% names(x))
  i <- which(i)[1]
  lista_wiki[[i]]
}

tabela_geral <- sapply(links_temporadas, FUN = tabela, USE.NAMES = FALSE)

sapply(tabela_geral, ncol)
#>  [1] 12 12 12 12 12 12 13 13 13 13 13 13 13 13 13 13 13 13 13 13
#sapply(tabela_geral, names)

common_names <- Reduce(intersect, lapply(tabela_geral, names))
tabela_reduzida <- lapply(tabela_geral, `[`, common_names)

tabela_final <- do.call(rbind, tabela_reduzida)
head(tabela_final)
#> # A tibble: 6 x 12
#>    Pos. Equipes       P         J     V     E     D    GP    GC SG      `%`
#>   <int> <chr>         <chr> <int> <int> <int> <int> <int> <int> <chr> <int>
#> 1     1 Cruzeiro      100      46    31     7     8   102    47 +55      72
#> 2     2 Santos        87       46    25    12     9    93    60 +33      63
#> 3     3 São Paulo     78       46    22    12    12    81    67 +14      56
#> 4     4 São Caetano   742      46    19    14    13    53    37 +16      53
#> 5     5 Coritiba      73       46    21    10    15    67    58 +9       52
#> 6     6 Internacional 721      46    20    10    16    59    57 +2       52
#> # ... with 1 more variable: `Classificação ou rebaixamento` <chr>

^{Created on 2022-04-03 by the reprex package (v2.0.1)}

To have all columns, including the "M" columns:

data.table::rbindlist(tabela_geral, fill = TRUE)