R fastdummies equivalent in sparkR

user_4544413 0
Viewed 145

I have a Spark dataframe with the following data: 

categories
1       John
2       Luis
3       Dora

For which I need to create a one hot ending version as: 

  categories categories_Dora categories_John categories_Luis
1       John               0               1               0
2       Luis               0               0               1
3       Dora               1               0               0

This is the current code I have: 

test <- data.frame("SN" = 1:2, "Age" = c(21,15), "Name" = c("John;Luis","Dora"))
df <- as.DataFrame(test)
df_2 = selectExpr(df, "split(Name, ';') AS categories","Name")


dat <- df_2 %>% 
 mutate(categories=explode(df_2$categories)) %>%
 select("categories")

The current solution I have is to convert this to a regular R dataframe, and apply the fast dummies function. Which works for this case but it wont´t work properly for a large dataset: 

r_df = dat %>% 
       SparkR::collect()
dummy_r = dummy_cols(r_df)

How can I get the same result using sparkR dataframes?

EDIT: I can not use sparklyr only sparkR

sparkr r character-encoding
1 Answers

It can be done with Sparklyr which has many of the feature transformer functions exposed.

library(sparklyr)

test <- data.frame("categories" = c("John", "Luis","Dora"))

sc <- sparklyr::spark_connect(master = "local")

d_tbl <- copy_to(sc, test, overwrite = TRUE)

d_tbl %>%
  ft_string_indexer(input_col = "categories", output_col = "cat_num") %>%
  mutate(cat_num = cat_num + 1) %>%
  ft_one_hot_encoder("cat_num", "cat_onehot") %>%
  sdf_separate_column("cat_onehot", 
                      paste("categories", pull(., categories), sep="_")) %>%
  select(-cat_num, -cat_onehot)

The output:

# Source: spark<?> [?? x 4]
  categories categories_John categories_Luis categories_Dora
  <chr>                <dbl>           <dbl>           <dbl>
1 John                     0               0               0
2 Luis                     0               1               0
3 Dora                     0               0               1

The ft_string_indexer generates a column names cat_num which has the a numeric value for each category. Very similar to as.numeric(factor) in R. The +1 is just to have the indexes from 1 to N. ft_one_hot_encoder does the magic at Spark level, the function return a vectorised value like a list with the encoding. The function sdf_separate_column expands the encoding to columns. The paste generates the colnames using the category levels. The select drops unnecessary columns used in the transformation.

user_3771180
Related