Using rvest (or another R package) to detect when the start of an HTML paragraph is a different format (e.g. emboldened)

I've read your related questions here on SO. Interesting work! I believe the solution is somewhere along the lines of:

1: Extract the relevant words from the HTML by doing what you're already doing

relevant_words <- rvest::html_text(your_raw_html_here)
# split word by word
relevant_words <- stringr::str_split(relevant_words, pattern = " ")

2: For every word, you'll have to find it in the raw HTML and check if it's inside a tag (say, a bold tag). You'll need to write regex for those. You'll also need to keep track of repeated ocurrences of a word and differentiate between them.

3: If you manage to do steps 1 and 2, you'll have a list that tells you the tags associated with every word found in step1. I believe that would be very much like How to apply rvest to a dataframe column of HTML to make a column of extracted emboldened words.

4: Now you need to check if the first N elements in your list have a major common style tag or not and compare that with the major style tag found in the rest of the paragraph.

I haven't done enough testing with this yet but the principle is to catch the cases you mentioned by looking at a child tag within the $raw for a given row.

Currently it is set-up to see if the the child's tag/type is in a given vector c("b", "strong", "i"); additionally, there is an additional test on the style attribute html_attr("style") %>% str_detect("italic|bold"). Not N/A will be output if attribute match not found.

The result of these two tests is as to whether this row should be considered to have a header (and need later splitting); boolean logic is used and wrapped in if_else to output false where child wasn't found. if_else(is.na(node), F, as.logical(style_flag | tag_flag). The result is written to a new column called header.

To finesse this a little, a comparison is then done on this new column and the bold_proportion_column to determine if ultimately this should be considered to contain a header row to be processed. The determination is currently on whether bold_proportion_column >= .5 OR header == T then T else F. Obviously, one would need to account for the italic_proportion_column at some point, or re-factor to include both in the proportion function (would be my choice). This determination is written to an outcome column.

I think there is a lot of work to be done on refining this approach and quality assuring it e.g. 1) I would want to re-work so we only parse the html once and 2) return the split of the child html from the rest to meet your requirement to split header section from rest across different rows; 3) I would also want to work through a pre-determined list of test cases from different pages to determine the validity of this approach; 4) I would re-factor to make functions such as get_child more generic e.g. pass in the css selector pattern for more versatility; elsewhere, pass in the proportion to be >= in outcome determination.

But it may be useful as a starter for 10....... I will likely re-visit this and refine if I can.

library(rvest)
library(tidyverse)
library(edgarWebR)
library(stringr)

paragraph_style_proportion <- Vectorize(function(html, css_selector) {
  missing_html_flag <- tryCatch(read_html(html), error = function(err) "NOT HTML")
  if (missing_html_flag == "NOT HTML") {
    style_proportion <- -1
  }
  else {
    page <- read_html(html)
    whole_paragraph_length <- nchar(str_squish(page %>% html_text()))
    style_text_length <- sum(nchar(str_squish(page %>% html_nodes(css_selector) %>% html_text())))
    style_proportion <- round(style_text_length / whole_paragraph_length, 2)
  }
  return(style_proportion)
})

get_child <- function(html) {
  child <- html %>%
    read_html() %>%
    html_node("*:nth-child(2)")
  return(child)
}

is_header <- function(node) {
  style_flag <- node %>%
    html_attr("style") %>%
    str_detect("italic|bold") %>% 
    if_else(is.na(.), F, .)
  tag_type <- node %>% html_name() %>% tolower()
  tag_flag <- tag_type %in% c("b", "strong", "i") # test if of certain type then classify
  flag <- if_else(is.na(node), F, as.logical(style_flag | tag_flag))
  return(flag)
}

df <- parse_filing("https://www.sec.gov/Archives/edgar/data/1070154/000114036109028731/form10k.htm", include.raw = TRUE)

df <- df %>%
  mutate(bold_proportion = paragraph_style_proportion(raw, "b, strong, [style*=bold]"))

df$header <- lapply(df$raw, function(x){is_header(get_child(x))})
df$outcome <-  with(df, ifelse(bold_proportion >=.5 | header == T, T, F))