sparklyr convert double to character

user_4332914 0
Viewed 271

I am encountering very weird behavior when converting doubles to characters in sparlyr. It seems that periods are added randomly. Here a reproducible example:

my_test_df <- data.frame(char_val = 004545, char_val2 = 100286908074)
my_test_spark <-  my_test_df %>%  copy_to(sc, ., 'my_test_df_spark', overwrite = TRUE)

my_test_spark

## Source: spark<my_test_df_spark> [?? x 2]
##  char_val    char_val2
##     <dbl>        <dbl>
##     4545 100286908074

my_test_spark %>%  
  mutate(char_val = lpad(as.character(char_val), 6, "0"),
         char_val2 = lpad(as.character(char_val2), 13, "0")) %>% 
  head 

## Source: spark<?> [?? x 2]
##  char_val char_val2    
##  <chr>    <chr>        
## 4545.0   1.00286908074

I really do not get why I have random periods in the final strings. Is there a way to avoid it?

sparklyr csv string
1 Answers

as.character(char_val2) is getting the value in scientific notation. lpad is truncating the scientific notation.

my_test_spark %>%  
  mutate(char_val3 = as.character(char_val2))
#> # Source: spark<?> [?? x 3]
#>   char_val    char_val2 char_val3       
#>      <dbl>        <dbl> <chr>           
#> 1     4545 100286908074 1.00286908074E11

For the .0 in the first column, it looks like as.character on a tbl_spark double column will add .0. You can convert to an integer to prevent this.

library(bit64)

my_test_spark %>%  
  mutate(char_val = lpad(as.character(as.integer(char_val)), 6, "0"),
         char_val2 = lpad(as.character(as.integer64(char_val2)), 13, "0"))
#> # Source: spark<?> [?? x 2]
#>   char_val char_val2    
#>   <chr>    <chr>        
#> 1 004545   0100286908074

Note that char_val2 needs to be a 64 bit integer.

user_17519610
Related