web-scraping: web-scraped object doesn't match information on the website and crashes RStudio

user_11883658 0
Viewed 142

I collected a series of URLs similar to this one. For each URL, I am using the rvest package to web-scrape information related to the address of every practitioner listed in each box of the webpage. By inspecting the HTML structure of the webpage, I could notice that the information I am trying to retrieve is present inside the HTML division called unit size1of2 (which appears, by hovering with the cursor, as div.unit.size1of2). Then, I used the following code to extract the information I need:

library(rvest)
library(xlm2)

webpage <- read_html(x = "myURL")

webpage_name <- webpage %>%
  html_nodes("div.unit.size1of2") %>%
  html_text(trim = T)

However, when I extract the information, the result I get it's super messy. First of all, there are information I didn't want to scrape, some of them seems to not even be present on the website. In addition, my RStudio IDE freezes for a while, and every time I try to output the result, without working properly afterwards with any command. Finally, the result is not the one I was looking for.

Do you think this is due to some kind of protection present on the website?

Thank you for your help!

rvest web-scraping division r html
1 Answers

You can start iterating on rows which can be selected using div.search-result .line and then :

  • getting the name using div:first-child h3
  • getting the ordinal using div:first-child p
  • getting the location by iterating on div:nth-child(2) p since there can be multiple locations (one has 5 locations on your page) and store them in a list

It's necessary to remove the tabs and new lines using gsub("[\t\n]", "", x) for the name and ordinal. For the addresses, you can get the text and split according to new line \n, remove duplicates new line and strip the first and last line to have a list like :

[1] "CABINET VÉTÉRINAIRE DV FEYS JEAN-MARC"
[2] "Cabinet Veterinaire"                  
[3] "ZA de Kercadiou"                      
[4] "XXXXX"                                
[5] "LANVOLLON"                            
[6] "Tél : 0X.XX.XX.XX.XX"

The following code also converts the list of vectors to a dataframe with all the data on that page :

library(rvest)
library(plyr)

url = "https://www.veterinaire.fr/annuaires/trouver-un-veterinaire-pour-soigner-mon-animal.html?tx_siteveterinaire_general%5B__referrer%5D%5B%40extension%5D=SiteVeterinaire&tx_siteveterinaire_general%5B__referrer%5D%5B%40vendor%5D=SiteVeterinaire&tx_siteveterinaire_general%5B__referrer%5D%5B%40controller%5D=FrontendUser&tx_siteveterinaire_general%5B__referrer%5D%5B%40action%5D=search&tx_siteveterinaire_general%5B__referrer%5D%5Barguments%5D=YToxOntzOjY6InNlYXJjaCI7YTo1OntzOjM6Im5vbSI7czowOiIiO3M6NjoicmVnaW9uIjtzOjA6IiI7czoxMToiZGVwYXJ0ZW1lbnQiO3M6MDoiIjtzOjU6InZpbGxlIjtzOjA6IiI7czoxMjoiaXRlbXNQZXJQYWdlIjtzOjI6IjEwIjt9fQ%3D%3D21a1899f9a133814dfc1eb4e01b3b47913bd9925&tx_siteveterinaire_general%5B__referrer%5D%5B%40request%5D=a%3A4%3A%7Bs%3A10%3A%22%40extension%22%3Bs%3A15%3A%22SiteVeterinaire%22%3Bs%3A11%3A%22%40controller%22%3Bs%3A12%3A%22FrontendUser%22%3Bs%3A7%3A%22%40action%22%3Bs%3A6%3A%22search%22%3Bs%3A7%3A%22%40vendor%22%3Bs%3A15%3A%22SiteVeterinaire%22%3B%7D7cd75ca141359a98763248c24da8103293a53d08&tx_siteveterinaire_general%5B__trustedProperties%5D=a%3A1%3A%7Bs%3A6%3A%22search%22%3Ba%3A5%3A%7Bs%3A3%3A%22nom%22%3Bi%3A1%3Bs%3A6%3A%22region%22%3Bi%3A1%3Bs%3A11%3A%22departement%22%3Bi%3A1%3Bs%3A5%3A%22ville%22%3Bi%3A1%3Bs%3A12%3A%22itemsPerPage%22%3Bi%3A1%3B%7D%7D86c9510d17c093c44d053714ab20567929a45f9d&tx_siteveterinaire_general%5Bsearch%5D%5Bnom%5D=&tx_siteveterinaire_general%5Bsearch%5D%5Bregion%5D=&tx_siteveterinaire_general%5Bsearch%5D%5Bdepartement%5D=&tx_siteveterinaire_general%5Bsearch%5D%5Bville%5D=&tx_siteveterinaire_general%5Bsearch%5D%5BitemsPerPage%5D=100&tx_siteveterinaire_general%5B%40widget_0%5D%5BcurrentPage%5D=127&cHash=8d8dc78e004b4b9d0ecfdf9b884f54ca"

rows <- read_html(url) %>%
  html_nodes("div.search-result .line")

strip <- function (x) gsub("[\t\n]", "", x)

i <- 1
data = list()
for(r in rows){
    addresses = list()
    j <- 1
    locations = r %>% html_nodes("div:nth-child(2) p")
    for(loc in locations){
        addresses[[j]] <- loc %>% html_text() %>% 
            gsub("[\t]", "", .) %>%  #remove tabs
            gsub('([\n])\\1+', '\\1', .) %>% #remove duplicate \n
            gsub('^\n|\n$', '', .) %>% #remove 1st and last \n
            strsplit(., split='\n', fixed=TRUE) #split by \n
        j <- j + 1 
    }
    data[[i]] <- c(
        name = r %>% html_nodes("div:first-child h3") %>% html_text() %>% strip(.),
        ordinal = r %>% html_nodes("div:first-child p") %>% html_text() %>% strip(.),
        addresses = addresses
    )
    i <- i + 1
}

df = rbind.fill(lapply(data,function(y){as.data.frame(t(y),stringsAsFactors=FALSE)}))

#show data
print(df)
for(i in 1:3){
    print(paste("name",df[i,"name"]))
    print(paste("ordinal",df[i,"ordinal"]))
    print(paste("addresses",df[i,"addresses"]))
    print(paste("addresses1",df[i,"addresses1"]))
    print(paste("addresses2",df[i,"addresses2"]))
    print(paste("addresses3",df[i,"addresses3"]))
}
user_26143640
Related