Why does my SelectorGadget does not select certain nodes while Webscraping with R

RGS 0
Viewed 16

So I have made code below in R for webscraping in given site - while in other sites this code works (after of course selecting the needed nodes), here it does not return anything. It seems that site itself when opening it at first does not allow to select such things even with mouse pointer. So I am wondering how to bypass this (as it happens in some other sites too).

install.packages('rvest')
install.packages('stringr')
install.packages('magrittr')
install.packages('tidyverse')
library(rvest)
library(stringr)
library(magrittr)
library(tidyverse)

#Pirmais
url_base <- "https://alkoutlet.lv/dzerieni/stiprie/rums.html?page="
l_out <- 2
urls <- paste0(url_base, seq(1, by = 1, length.out = l_out))
urls
# Helper function for parsing overview


parse_overview <- function(x){
  tibble(
    title = html_text(html_nodes(x, '.ProductCard-Name_isLoaded'), TRUE),
    price = html_text(html_nodes(x, '.ProductCard-PriceWrapper'), TRUE),
    description = html_text(html_nodes(x, '.ProductCard-ShortSpecification'), TRUE),
    link = str_trim(html_attr(html_nodes(x, '.ProductCard-Name_isLoaded'), 'href'))%>%paste("https://alkoutlet.lv",.,sep=""))
}

# Scrape overview    
Result <- urls %>% 
  map(read_html) %>% 
  map_df(parse_overview)

View(Result)

This is for first 2 pages from the section in the site about rum - I'm trying to scrape prices, description and name (and also link, but I am not sure if I have chosen the correct node).

Does anyone has any ideas how to make it work? It seems the site does not offer nodes to select when you open it at first, so it might be some type of prevention for this - so how can it be bypassed?

web-scraping r
1 Answers

The information is not loaded as HTML. I suggest this method instead with httr2 to request the data as the site does. The price information is in a nested dataframe price_range.

library(tidyverse)
library(httr2)

'https://alkoutlet.lv/graphql?hash=2951167027&sort_1={"name":"ASC"}&filter_1={"price":{},"category_id":{"eq":13},"customer_group_id":{"eq":"0"}}&pageSize_1=24&currentPage_1=2&_currency=""' %>% 
  request() %>%  
  req_perform() %>%  
  resp_body_json(simplifyVector = TRUE) %>% 
  .$data %>% 
  .$products %>% 
  .$items %>% 
  as_tibble()

# A tibble: 24 x 23
      id sku    name     type_id stock~1 volum~2 alco_~3 sugge~4 categ~5 goodW~6 price~7 thumb~8
   <int> <chr>  <chr>    <chr>   <chr>     <int>   <int> <chr>   <list>  <lgl>     <dbl> <chr>  
 1   642 366419 Rums Co~ simple  IN_STO~      82      83 NA      <df>    NA         1.76 /imp/o~
 2   634 366433 Rums Co~ simple  IN_STO~      82      83 NA      <df>    NA         1.26 /imp/o~
 3   631 366443 Rums Co~ simple  IN_STO~      82      83 NA      <df>    NA         1.26 /imp/o~
 4   672 366310 Rums Co~ simple  IN_STO~      82     813 NA      <df>    NA         1    /imp/o~
 5  3148 366584 Rums Da~ simple  IN_STO~      82     835 NA      <df>    NA         4    /3/6/3~
 6  3147 366589 Rums Da~ simple  IN_STO~      82     835 NA      <df>    NA         4    /3/6/3~
 7  2644 364746 Rums De~ simple  IN_STO~      82     835 672     <df>    NA         3    /3/6/3~
 8  2595 366565 Rums De~ simple  IN_STO~      82     894 672     <df>    NA         3    /3/6/3~
 9  2896 364747 Rums De~ simple  IN_STO~      82     835 672     <df>    NA         3    /3/6/3~
10   810 362939 Rums De~ simple  IN_STO~      82      83 NA      <df>    NA         4    /3/6/3~
# ... with 14 more rows, 17 more variables:
#   price_range$minimum_price$discount$percent_off <dbl>,
#   price_range$minimum_price$final_price <df[,2]>, $$final_price_excl_tax <df[,2]>,
#   $$regular_price <df[,2]>, $$regular_price_excl_tax <df[,2]>, thumbnail$url <chr>,
#   small_image <df[,2]>, short_description <df[,1]>, stock_item <df[,2]>,
#   special_from_date <chr>, special_to_date <chr>, price_tiers <list>, attributes <list>,
#   url <chr>, review_count <int>, rating_summary <int>, mp_label_data <list>, and ...
# i Use `print(n = ...)` to see more rows, and `colnames()` to see all variable names
Tom Hoel0
Related