R: scraping authors name from table of contents of Sage journal

user_9731332 0
Viewed 93

I’m trying to scrape the table of content of a journal published by Sage. Scraping titles and URLs is a no brainer. Scraping author names is tricky though, maybe because they open a pop-up with lots of information (affiliation, orcid, etc.), and SelectorGadget seems unable to decipher between all these. After several attempts, the following code:

author_1 <- read_html("https://journals.sagepub.com/toc/ossa/40/1") %>%
  html_nodes('.all') %>%
  html_text(trim = TRUE)
author_1

gives:

[1] "Andrew D. Brown Andrew D. BrownUniversity of Bath, UK View ORCID profileSee all articles by this author\nSearch Google Scholar\n for this author"
[2] "Peter Fleming Peter FlemingSee all articles by this author\nSearch Google Scholar\n for this author"
[3] "Mike Reed Mike ReedCardiff Business School, UKSee all articles by this author\nSearch Google Scholar\n for this author, Gibson Burrell Gibson BurrellUniversities of Leicester and Manchester, UK View ORCID profileSee all articles by this author\nSearch Google Scholar\n for this author"

etc.

Regex to clean this is beyond my limited skills (especially since some articles, like the 3rd one, have several authors). Any help would be greatly appreciated.

rvest r
1 Answers

You can identify the parent nodes and then map over these to return a list that keeps the authors together:

library(rvest)
library(purrr)

page <- read_html("https://journals.sagepub.com/toc/ossa/40/1")

page %>%
  html_nodes('div.tocAuthors') %>%
  map(~ html_nodes(.x, 'div.header a.entryAuthor') %>%
      html_text(trim = TRUE))

[[1]]
[1] "Andrew D. Brown"

[[2]]
[1] "Peter Fleming"

[[3]]
[1] "Mike Reed"      "Gibson Burrell"

...

Or for a single string of authors per article:

page %>%
  html_nodes('div.tocAuthors') %>%
  map_chr(~ html_nodes(.x, 'div.header a.entryAuthor') %>%
         html_text(trim = TRUE) %>% toString)

 [1] "Andrew D. Brown"                              "Peter Fleming"                               
 [3] "Mike Reed, Gibson Burrell"                    "Joep Cornelissen"                            
 [5] "Silviya Svejenova"                            "Mélodie Cartel, Eva Boxenbaum, Franck Aggeri"
 [7] "Deborah N. Brewis"                            "Renate Ortlieb, Barbara Sieben"              
 [9] "Lynne Andersson, Dirk Lindebaum, Mar Pérezts" "Lidia Greco"                                 
[11] "Michael Rowlinson"                            "Andrew Crane"                                
[13] "Jean Jenkins"
user_28352610
Related