I have 100 million rows stored in many .csv files in a distributed file system. I'm using spark_read_csv() to load the data without issue. Many of my columns are stored as character logical values: "true", "false", "<na>". I do not have control over this.
When I attempt to convert the values to logical, the "<na>" values are converted to FALSE with the "false" values. Any thoughts on how to overcome this?
test_lgl <-
tibble(a = c(TRUE, TRUE, NA, NA, FALSE, FALSE),
b = c("true", "true", "na", "<na>", "false", "f"))
test_lgl %>% mutate_if(is.character, as.logical)
# this works
a b
<lgl> <lgl>
1 TRUE TRUE
2 TRUE TRUE
3 NA NA
4 NA NA
5 FALSE FALSE
6 FALSE NA
sc <- spark_connect(master = "local")
spark_lgl <- copy_to(sc, test_lgl)
spark_lgl %>% mutate_if(is.character, as.logical)
# this does not
a b
<lgl> <lgl>
1 TRUE TRUE
2 TRUE TRUE
3 FALSE FALSE
4 FALSE FALSE
5 FALSE FALSE
6 FALSE FALSE