I am trying to scrape hover text content from some plotly traces published on the web. I have not performed this type of scraping before and am trying to do this in R without selenium or phantomjs if possible... perhaps using V8? I was wondering if someone could point me in the right direction. Link to plots are below. Specifically looking for data in plots from Figure 21: Positivity rate for COVID-19 in Alberta by zone. Thanks! https://www.alberta.ca/stats/covid-19-alberta-statistics.htm

R web scraping plotly trace hover text without selenium or phantomjs

Using rvest and jsonlite the following code will get you the data you are looking for. The data for the plot.ly diagrams is stored in <script> tags.

First step is to identify the widget ID of the figure of interest, the code below shows you how to find the widget ID by looking for the caption text of the figure of interest. Then you can search for the correct node with html_nodes() and html_attrs(). jsonlite::fromJSON() converts the JSON data to an R list object.

library(rvest)
library(jsonlite)
library(purrr)
library(stringr)
library(dplyr)


url <-
  "https://www.alberta.ca/stats/covid-19-alberta-statistics.htm#laboratory-testing"

raw_html <- read_html(url)

# get widget ID

caption <-
  "Figure 21: Positivity rate for COVID-19 in Alberta by zone."

figure_divs <- html_nodes(raw_html, ".figure")

figure_21_div_lgl <- grepl(caption, figure_divs)

widget_id <-
  figure_divs[figure_21_div_lgl] %>%
  html_nodes("div") %>%
  html_attr("id")

# find data for the correct widget_id

data_for <-
  html_nodes(raw_html, "script") %>%
  html_attr("data-for")

data_for_figure_21_lgl <-
  !is.na(data_for) & data_for == widget_id

data_for_figure_21 <-
  html_nodes(raw_html, "script") %>%
  .[data_for_figure_21_lgl] %>%
  html_text()

dff21_l <- fromJSON(data_for_figure_21)

In order to extract the data shown in the tooltips (“hover texts”) we need to iterate through the different elements. First extracting the DOM structure with html_text(). After that we extract the text with html_text(). We iterate through the elements a few more times to split and clean the strings, so that we can finally convert the results into a data.frame.

tooltip_text_raw <- unlist(dff21_l$x$data$text)
tooltip_text <- map(tooltip_text_raw, read_html)
tooltip_text <- map(tooltip_text, html_text) %>% unlist()

tooltip_text_split <- strsplit(tooltip_text, "\\:")

tooltip_text_split_almost_clean <-
  map(tooltip_text_split,
      ~ gsub("Report Date|Percent|Number of tests", "", .x))

tooltip_text_split_clean <-
  map(tooltip_text_split_almost_clean, ~ str_squish(.[. != ""]))

tests_df <-
  map_dfr(tooltip_text_split_clean,
          ~ data.frame(
            date = as.Date(.x[1]),
            percent = .x[2],
            tests = .x[3]
          ))

head(tests_df)
#>         date percent tests
#> 1 2020-03-06    9.68    31
#> 2 2020-03-07    0.00   142
#> 3 2020-03-08    0.00   213
#> 4 2020-03-09    2.51   239
#> 5 2020-03-10    3.90   282
#> 6 2020-03-11    1.05   572

R web scraping plotly trace hover text without selenium or phantomjs

1 Answers