Use rvest to link single high level categories to multiple items within that category

user_11631012 0
Viewed 36

Firstly, apologies if the title of this question isn't clear - let me try to explain. I'm scraping some data from www.bbc.co.uk/iplayer. The site is structured in such as way that there are approx 10 categories (e.g. sports, drama, popular now), within which there are approximately 10 programmes. I've managed to scrape most of the data I need (programme title, genre, and synopsis) and I know how to combine this into a new data frame. Here's my code so far:

df <- read_html("https://www.bbc.co.uk/iplayer")

title <- df %>%
    html_nodes("div.content-item__title.typo.typo--skylark.typo--bold") %>%
    html_text()
genre <- df %>%
    html_nodes("div.content-item__labels") %>%
    html_text()
synopsis <- df %>%
    html_nodes("div.content-item__info__primary") %>%
    html_text()

### Combine into a tabble/tibble ###
df2 <- tibble(title=title,
              genre=genre,
              synopsis=synopsis)

The problem is that "category" exists at a higher level and there are only around 10 of these (whereas there are 110ish individual programmes/genres/synopses).

I know the code to get the categories:

category <- df %>%
    html_nodes("div.section__header") %>%
    html_text()

However, this only returns approx 10 categories, and therefore it can't be combined into a table with the other values. What I want is a final table that also includes the category - e.g. if there are 10 titles in the first category, and that category is called "Most Popular", then in the category column is should say "Most Popular" for all 10 of those titles.

Hope that's all clear. Happy to elaborate if necessary.

rvest web-scraping r
1 Answers

You could use purrr::map_dfr to apply a function to each section node. That function would return a tibble, in this case, of the required info for each section. map_dfr would then map these tibbles to a final DataFrame. Within the tibble, the shorter header would be recycled to match length of the other columns.

library(rvest)
library(purrr)
library(dplyr)

get_section_data <- function(section){
  
  t <- tibble(
    header = section %>% html_node('h2') %>% html_text(),
    title = section %>% html_nodes('.content-item__title') %>% html_text(),
    genre = section %>% html_nodes('.content-item__labels') %>% html_text(),
    synopsis = section %>% html_nodes('.content-item__info__primary') %>% html_text()
  )
  return(t)
}

page <- read_html('https://www.bbc.co.uk/iplayer')
sections <- page %>% html_nodes('.section__content')
results <- purrr::map_dfr(sections, get_section_data)
user_62412350
Related