scraping data from web page within <div> tag with R

user_16477195 0
Viewed 58

I would like to scrape product name and the rating from a webpage. Upon inspecting the element, I know I need to get the data from product__title and attraqt-star-rating-stars__bar. But I am not sure how to do it as this is embedded within the multiple layers of tag. I've tried the following with no avail; any suggestions are welcome.

library(rvest)
library(dplyr)
url = 'https://www.chemistwarehouse.com.au/shop-online/159/oral-hygiene-and-dental-care'
stores <- read_html(url) 

stores %>% html_nodes('body') %>% 
  html_nodes('.product__title') %>% 
  rvest::html_text()

stores %>% html_nodes('body') %>% 
  html_nodes('attraqt-star-rating-stars__bar') %>% 
  rvest::html_text()
rvest web-scraping r
1 Answers

Data is pulled dynamically from an API call. As the json returned is nested you need to extract the desired info e.g., by writing a couple of user-defined functions.

I first extract the listings (list of products), then have a function get_info, which takes an individual product listing and extracts the title and rating and returns a tibble. As the index at which the rating may appear can vary, I have an additional helper function get_rating_index, which retrieves dynamically the correct index for the rating. This function passes the index back to get_info.

I apply get_info over the list of product info, listings, using map_dfr to generate a final DataFrame from each tibble.

library(jsonlite)
library(purrr)
library(dplyr)

data <- jsonlite::read_json("https://www.chemistwarehouse.com.au/searchapi/webapi/search/category?category=159&index=0&sort=")

listings <- data$universes$universe[[1]]$`items-section`$items$item

get_info <- function(listing) {
  tibble(
    title = listing$attribute[[2]]$value[[1]]$value,
    rating = listing$attribute[[get_rating_index(listing$attribute)]]$value[[1]]$value %>% as.numeric()
    ) -> t
  return(t)
}

get_rating_index <-function(attribute){
  return(match(T, map(attribute, ~{.x$name == 'bv_star_rating'})))
}

dental_product_ratings <- purrr::map_dfr(listings, get_info)
user_62412350
Related