warning xml_find_all.xml_node while webscrape with rvest

user_4465634 0
Viewed 787

I want to scrape data from this link. I used rvest and a loop to get the tables.

This is my code:

require(dplyr)
require(rvest)

# store web url
url <- "https://corona.thueringen.de/covid-19-bulletin/"
# check xpath
xpath_part1="/html/body/main/div[2]/div/section[2]/ul/li["
xpath_part2="]/div/div/div[2]/div/table"

# save tables
for(i in 1:50){
  tbl_test_ <- 
    url %>% 
    xml2::read_html() %>%
    rvest::html_nodes(xpath=paste0(xpath_part1, i, "]", xpath_part2)) %>%
    rvest::html_table(fill = TRUE) %>%
    dplyr::rename(Gesundheitsamt=1)
}

I get the following error:

Fehler in UseMethod("rename_") : 
  nicht anwendbare Methode für 'rename_' auf Objekt der Klasse "list" angewendet
Zusätzlich: Warnmeldung:
In xml_find_all.xml_node(x, make_selector(css, xpath)) :
  Invalid expression [1207]

edit 1:

I have cleaned my code:

# save tables
for(i in 1:50){
  tbl_test_ <- 
    url %>% 
    xml2::read_html() %>%
    rvest::html_nodes(xpath=paste0(xpath_part1, i, xpath_part2)) %>%
    rvest::html_table(fill = TRUE) %>%
    dplyr::rename(Gesundheitsamt=1)
}

and get the error message:

Fehler in open.connection(x, "rb") : HTTP error 404.

Any suggestions on how to get correct for this error and get the tables?

Edit 2: At the current table can I add some additional informations. I get this with:

# store web url
url <- "https://corona.thueringen.de/bulletin"
#Read the page
page <- url %>%  read_html()

fxp <- "/html/body/main/div[2]/div/section[1]/div[2]/div[1]/div[2]/div"

tbl_current <- 
  page %>% 
  html_nodes("table[align=left]") %>% 
  html_table() %>% 
  .[[1]] %>%
  dplyr::rename(Gesundheitsamt=1) %>% 
  dplyr::mutate(note_1=rvest::html_text(rvest::html_nodes(th_bulletin, xpath=paste0(fxp, "/div[1]/h2")))) %>% 
  dplyr::mutate(note_2=rvest::html_text(rvest::html_nodes(th_bulletin, xpath=paste0(fxp, "/div[3]/h3"))))

To work with the full xpath is not the best way, but I didnt know to get the nodes only from a specific part from the site (/html/body/main/div[2]/div/section1).

For the archived tables I'm looking also for this informations. I tryed to store it at first in a list. But this seems to works only for one html_node. 

tbl_all <-
page %>%
  rvest::html_nodes("table[align=left]")

Here I get a list, with one element for each table

[[1]] table
[[2]] table
.
.
.

To use more than one node is not working.

tbl_all_ <-
  page %>%
  rvest::html_nodes("table[align=left]", "h2", "h3")

It is possible to get more then one kind of node in a list? I want to get

[[1]] for /html/body/main/div[2]/div/section[2]/ul/li[1]
table 1
table 2
h2
h3
[[2]] for /html/body/main/div[2]/div/section[2]/ul/li[2]
table 1
table 2
h2
h3
[[3]] for /html/body/main/div[2]/div/section[2]/ul/li[3] ...

Thanks in advance.

rvest web-scraping r for-loop
1 Answers

Here is an easier method to retrieve the requested tables.
Instead of retrieving each table individually, read the page once (much faster and less web traffic) and the parse all of the tables using html_nodes with the CSS 'table' selector.
This will return all 142 tables on the page. It looks like the tables are in groups of 3. By specifying the desired attribute with the CSS selector this list could be reduced to provide the desired subset.

require(dplyr)
require(rvest)

# store web url
url <- "https://corona.thueringen.de/bulletin"
#Read the page
page <- url %>%  read_html() 
#retrieve all of the tables
tables<- page %>% rvest::html_nodes("table")

#cycle through the selected tables
#and stored as needed
for (table in tables) {
  print(table %>% html_table())
  Sys.sleep(2)
}

#another possible selection method:
# select tables where the align attribute = "left"
page %>% html_nodes("table[align=left]") %>% html_table()

Update
Since you are looking for all of the tables stored as a list within a list. Here is a revision. The page is broken down in a series of li with class="th-lst-itm card". First step is to retrieve this list of parent nodes and then parse the all of tables from each parent. 

# store web url
url <- "https://corona.thueringen.de/bulletin"
#Read the page
page <- url %>%  read_html() 

#find archived list items with class=th-lst-itm
listofitems<-page %>% rvest::html_nodes("li.th-lst-itm")

#store all of the tables in as list within a list
answer<-lapply(listofitems, function(item){
  item %>% html_nodes("table") %>% html_table()
})

# #first try but did not work for all elements
# names(answer)<-listofitems %>% html_node("h2") %>% html_text() %>% trimws()

#name the elements
names(answer)<-listofitems %>% html_node("span.th-lbl") %>% html_text() %>% trimws() 

#Pull a sample
answer[["COVID-19 / Bulletin der Thüringer Landesregierung 02/2020" ]]
user_57922440
Related