Sparklyr handling categorical variables
I came from R background and I am used to categorical variables being handled in the backend (as factor). With Sparklyr it is quite confusing using string_indexer or onehotencoder.
For example, I have a number of variables which has been encoded as numerical variables in the original dataset but they are actually categorical. I want to use them as categorical variables but am not sure I am doing it correctly.
library(sparklyr)
library(dplyr)
sessionInfo()
sc <- spark_connect(master = "local", version = spark_version)
spark_version(sc)
set.seed(1)
exampleDF <- data.frame (ID = 1:10, Resp = sample(c(100:205), 10, replace = TRUE),
Numb = sample(1:10, 10))
example <- copy_to(sc, exampleDF)
pred <- example %>% mutate(Resp = as.character(Resp)) %>%
sdf_mutate(Resp_cat = ft_string_indexer(Resp)) %>%
ml_decision_tree(response = "Resp_cat", features = "Numb") %>%
sdf_predict()
pred
The prediction from the model is not categorical. See below. Does it mean I also have to convert back from prediction to Resp_cat and then to Resp?
R version 3.4.0 (2017-04-21)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)
spark_version(sc)
[1] ‘2.1.1.2.6.1.0’
Source: table<sparklyr_tmp_74e340c5607c> [?? x 6]
Database: spark_connection
ID Numb Resp Resp_cat id74e35c6b2dbb prediction
<int> <int> <chr> <dbl> <dbl> <dbl>
1 1 10 150 8 0 8.000000
2 2 3 191 4 1 4.000000
3 3 4 146 9 2 9.000000
4 4 9 125 5 3 5.000000
5 5 8 107 2 4 2.000000
6 6 2 110 1 5 1.000000
7 7 5 133 3 6 5.333333
8 8 7 154 6 7 5.333333
9 9 1 170 0 8 0.000000
10 10 6 143 7 9 5.333333