Suppose I want to use ft_max_abs_scaler on every column of a dataset. This is what's in the documentation:
sc <- spark_connect(master = "local")
iris_tbl <- sdf_copy_to(sc, iris, name = "iris_tbl", overwrite = TRUE)
features <- c("Sepal_Length", "Sepal_Width", "Petal_Length", "Petal_Width")
iris_tbl <- iris_tbl %>%
ft_vector_assembler(input_col = features,
output_col = "features_temp") %>%
ft_max_abs_scaler(input_col = "features_temp",
output_col = "features")
Note that ft_vector_assembler creates a new column features_temp and ft_max_abs_scaler creates another new column features. Now suppose I want to break down the vector into individual columns, I have to do this:
iris_tbl <- iris_tbl %>% sdf_separate_column("features", into = features)
# result in error because column name cannot be the same
Since there is no good way to delete columns, I wonder if there is a better way to do feature transformations with Sparklyr without keeping creating new columns.