as.numeric() producing NAs for what should be numbers

user_10415056 0
Viewed 95

I am scraping the Illinois department of corrections website (https://www2.illinois.gov/idoc/facilities/Pages/Covid19Response.aspx) and trying to create a vector called Staff.Confirmed for the number of staff who have contracted COVID-19. I run the following code


    url <- "https://www2.illinois.gov/idoc/facilities/Pages/Covid19Response.aspx"
    web <- read_html(url)
    Staff.Confirmed.html <- html_nodes(web,'.soi-rteTableOddCol-1:nth-child(2)')
    Staff.Confirmed <- html_text(Staff.Confirmed.html)
    Staff.Confirmed <- as.numeric(Staff.Confirmed)

which produces the following output when I call Staff.Confirmed:

[1] "​1"  "​1"  "​1"  "​4"  "​5"  "​7"  "​1"  "​1"  "​2"  "​1"  "​13" "​3"  "​4"  "​2"  "​4"  "​2"  "​0"  "​8"  "​8"  "​1"  "​79" "​37" "​0"  "​1"  "​1"

However, when I run

Staff.Confirmed <- as.numeric(Staff.Confirmed)

I get the warning message "NAs introduced by coercion." Every number has become an NA. As far as I can tell, there are no whitespaces, nor any other issues which should be causing this. Has anyone else encountered this issue before?

I tried running

Staff.Confirmed <- gsub(pattern="^[0-9]",replacement="",Staff.Confirmed)
Staff.Confirmed <- as.numeric(Staff.Confirmed)

But the same error occurred. Any help would be greatly appreciated! Thank you.

gsub na web-scraping r character-encoding
2 Answers

There is a unicode letter (<U+200B> = zero width space) hiding at the first position of each string. Remove it and it works:

url <- "https://www2.illinois.gov/idoc/facilities/Pages/Covid19Response.aspx"
web <- read_html(url)
Staff.Confirmed.html <- html_nodes(web,'.soi-rteTableOddCol-1:nth-child(2)')
Staff.Confirmed <- substr(html_text(Staff.Confirmed.html), 2, 999)
Staff.Confirmed <- as.numeric(Staff.Confirmed)

It was very easy to spot in the "environment pane" of RStudio. The pane displayed the vector like this:

chr [1:25] "<U+200B>1" "<U+200B>1" "<U+200B>1" "<U+200B>4" ...

and I was able to confirm it with nchar(html_text(Staff.Confirmed.html)) that each string was too long by exactly 1 character.

user_132415450

New idea. Two principles:

  1. This approach does not look for invalid characters. It looks for everything we want to keep and discards the rest.
  2. We get the table in one large gulp instead of column by column
    # Fetch the data
    url <- "https://www2.illinois.gov/idoc/facilities/Pages/Covid19Response.aspx"
    web <- read_html(url)
    
    # extract table and keep only valid characters
    raw <- as.character(html_node(web, ".soi-rteTable-1")) # get table and coerce it to character
    nchar(raw) # see how long it is
    raw <- gsub("[^[:alnum:]|[:space:]|[:punct:]]", "", raw) # keep only valid characters
    nchar(raw) # length got reduced
    df <- html_table(minimal_html(raw), header = TRUE, trim = TRUE)
    df <- df[[1]] # html_table returns a list, so get the first element

    # check if columns are numeric
    str(df)
    'data.frame':   26 obs. of  5 variables:
     $ Locations                         : chr  "Crossroads ATC" "Danville" "Dixon" "East Moline" ...
     $ Staff Confirmed                   : int  1 1 1 4 5 7 1 1 2 1 ...
     $ Staff Recovered                   : int  1 1 1 2 5 7 1 1 2 1 ...
     $ Incarcerated Individuals Confirmed: int  3 0 0 28 0 4 0 0 15 0 ...
     $ Incarcerated Individuals Recovered: int  3 0 0 1 0 4 0 0 15 0 ...

Voilà! On character column with labels and 4 integer columns.

user_132415450
Related