Scrape text mixed with emojis with rvest

Question

I'm trying to scrape live chats from Youtube streaming. I can use rvest's html_text() to get the texts and I can also use html_attr("shared-tooltip-text") to get the emojis. For my data, I can then combine them by putting the emojis behind the text to create a text string. HOWEVER, I can't figure out how to get the text and emojis in the order they appear in the original message.

Here is a sample chat message below



"text pt1"

 

"text pt2"

If I use

messages <- html_elements(chat_html,"#message")
texts <- html_text(messages)

I get

"text pt1 text pt2"

I can then use emoji <- html_attr(html_elements(messages,"img"),"shared-tooltip-text") to get the emojis' names.

But the fact that the emojis are in between the text is not captured.

Is there a way to scrape and get

"text pt 1 :yellow_heart: text pt 2 :blue_heart:"

Thank you very much! Sorry, I'm a complete amateur. TIA

user_19512611 · Answer

Here we can use an xpath the selects both text and img nodes, and a custom ifelse function to extract the correct information and collapse it:

library(rvest)

sample_html <- read_html(' text pt1  text pt2  ')

get_text <- function(x) {
  ifelse(html_name(x) == "text", html_text(x), html_attr(x, "shared-tooltip-text")) |>
    paste(collapse = " ")
}

sample_html |>
  html_elements(xpath = "//span[@id='message']//text() | //span[@id='message']//img[contains(@class, 'emoji')]") |>
  get_text()

#[1] [1] " text pt1  :yellow_heart:  text pt2  :blue_heart:"

Let me know if this works!

Edits upon further query:

sample_html <- read_html('

   "text 1"
    


 "text 2" ')

get_text <- function(x) {
  ifelse(html_name(x) == "text", html_text(x), html_attr(x, "alt")) |>
    paste(collapse = " ")
}

texts <- sample_html |>
  html_elements(xpath = "//span[@id='message']//text() | //span[@id='message']//img[contains(@class, 'emoji')]")

get_text(texts) %>%
  str_remove_all("\"|\n") %>%
  str_trim() %>%
  str_squish()

#[1] "text 1 :yellow_heart: text 2"

It is not a list, there are just some things you have to get rid of this time for it to look like a proper string.

Scrape text mixed with emojis with rvest

1 Answers