Scrape wiki collapsible table with rvest

user_8398165 0
Viewed 359

I have used revest to scrape wiki before but this time I can't gather the data from a collapsible table (first table of the post):

library(tidyverse)
library(rvest)

url <- "https://es.wikipedia.org/wiki/Anexo:Sondeos_de_intenci%C3%B3n_de_voto_para_las_elecciones_generales_de_Espa%C3%B1a_de_noviembre_de_2019"

sample <- url %>%
  read_html() %>%
  html_nodes(xpath = '/html/body/div[3]/div[3]/div[4]/div/table[1]') %>% 
  html_table(header = T)

I get this error message:

Error in if (length(p) > 1 & maxp * n != sum(unlist(nrows)) & maxp * n != : missing value where TRUE/FALSE needed

I have the feeling that this is due to the nested cells but I can't find a way to delete the first and last rows.

rvest web-scraping r
1 Answers

I am not quite sure, but rvest is maybe having trouble with the table due to the links in the header. One option is to manually read the rows and then create the desired dataframe. This is not a very elegant solution, but it gets there.

This table also has a few rows with an incomplete number of cells. These rows are only in the table's header and footer and are not needed. I removed these rows, converted to a matrix/data frame and then renamed a couple of the columns.
Please see the code's comments for more information.

library(rvest)
url <- "https://es.wikipedia.org/wiki/Anexo:Sondeos_de_intenci%C3%B3n_de_voto_para_las_elecciones_generales_de_Espa%C3%B1a_de_noviembre_de_2019"

page<- url %>% read_html() 
tables<- page%>% html_nodes('table') 

#first attempt
#tables[1] %>% html_table(fill=TRUE)   #Causes an error

#pull the rows
rows<-tables[1] %>% html_nodes('tr') 
#remove rows which do not have the full number cells
rows<- rows[-which(sapply(rows, function(x){length(html_children(x))}) != 20)]

#convert to text
values<-rows %>% html_nodes('td') %>% html_text() %>% trimws()
#create data frame
answer<-as.data.frame(matrix(values, ncol = 20, byrow=TRUE))

#find variable names and rename columns
variablenames <- rows %>% html_nodes('th') %>% html_text() %>% trimws()
names(answer)[which(variablenames !="")] <- variablenames[which(variablenames !="")]

answer

You will still need to manually rename the column with the actual TV station names.

user_57922440
Related