Scrape headlines and text (all p after h) in R

user_8833311 0
Viewed 39

I would like to scrape all headlines and the corresponding paragraph/text after a headline. My web document has several headlines. Finally, all text data should be ready for analysis in a dataframe. Headlines as h2 or belong to class mw-headline. I tried various sibling/children options, none worked so far.

library(rvest)
# define url
url <- "https://de.wikisource.org/wiki/Die_Gartenlaube_(1853)/Heft_1"
# download content
html_document  <- read_html(url)

html_document %>%
  # extract headlines
  html_nodes("h2") %>%
  # extract text
  html_text() -> title

html_document %>%
  # extract paragraphs
  html_nodes("p")  %>%
  # extract text
  html_text() -> text
rvest web-scraping r
1 Answers

Here is an approach that can be considered :

library(stringr)
url <- "https://de.wikisource.org/wiki/Die_Gartenlaube_(1853)/Heft_1"
raw_Html <- readLines("https://de.wikisource.org/wiki/Die_Gartenlaube_(1853)/Heft_1")
raw_Html <- paste0(raw_Html, collapse = "")
stringr::str_extract_all(raw_Html, pattern = 'mw-headline[^<]*</span>')

[[1]]
[1] "mw-headline\" id=\"An_unsere_Freunde_und_Leser!\">An unsere Freunde und Leser!</span>"
[2] "mw-headline\" id=\"Aus_der_Menschenheimath.\">Aus der Menschenheimath.</span>"        
[3] "mw-headline\" id=\"Anmerkungen_(Wikisource)\">Anmerkungen (Wikisource)</span>"
user_92456230
Related