Convert a string to logical in R with sparklyr

Viewed 935

I have 100 million rows stored in many .csv files in a distributed file system. I'm using spark_read_csv() to load the data without issue. Many of my columns are stored as character logical values: "true", "false", "<na>". I do not have control over this.

When I attempt to convert the values to logical, the "<na>" values are converted to FALSE with the "false" values. Any thoughts on how to overcome this?

test_lgl <- 
  tibble(a = c(TRUE, TRUE, NA, NA, FALSE, FALSE),
         b = c("true", "true", "na", "<na>", "false", "f"))

test_lgl %>% mutate_if(is.character, as.logical)

# this works
   a     b
  <lgl> <lgl>
1  TRUE  TRUE
2  TRUE  TRUE
3    NA    NA
4    NA    NA
5 FALSE FALSE
6 FALSE    NA

sc <- spark_connect(master = "local")
spark_lgl <- copy_to(sc, test_lgl)

spark_lgl %>% mutate_if(is.character, as.logical)

# this does not
      a     b
  <lgl> <lgl>
1  TRUE  TRUE
2  TRUE  TRUE
3 FALSE FALSE
4 FALSE FALSE
5 FALSE FALSE
6 FALSE FALSE
1 Answers
Related