sparklyr + lm with categorical predictors

Viewed 56

I'm trying to run lm on sparklyr with categorical predictors. An example that works in plain R fails in sparklyr:

# this works
lm(Petal.Length ~ as.factor(Species), data = iris)

# this fails
spark_apply(
    iris_tbl,
    function(e) broom::tidy(lm(Petal_Length ~ as.factor(Species), e)),
    names = c("term", "estimate", "std.error", "statistic", "p.value"),
    # group_by = "Species"
    )

I was trying to imitate this example and simply replaced independent variable to be categorical.

Error log:

...
21/06/09 22:48:01 ERROR sparklyr: RScript (3130) terminated unexpectedly: contrasts can be applied only to factors with 2 or more levels 
21/06/09 22:48:01 ERROR sparklyr: RScript (3130) collected callstack: 
16: stop("contrasts can be applied only to factors with 2 or more levels")
15: `contrasts<-`(`*tmp*`, value = contr.funs[1 + isOF[nn]])
14: model.matrix.default(mt, mf, contrasts)
13: model.matrix(mt, mf, contrasts)
12: lm(Petal_Length ~ as.factor(Species), e)
11: broom::tidy(lm(Petal_Length ~ as.factor(Species), e)) 
(21/06/09 22:48:01 INFO sparklyr: Session (3130) is shutting down with expected SocketException,java.net.SocketException: Socket closed)
21/06/09 22:48:01 ERROR sparklyr: Worker (3130) failed to complete R process
(21/06/09 22:48:01 ERROR sparklyr: Worker (3130) failed to run rscript: ,java.lang.Exception: sparklyr worker rscript failure with status 255, check worker logs for details.)
21/06/09 22:48:01 INFO sparklyr: Worker (3130) completed wait using lock for RScript
21/06/09 22:48:01 ERROR Executor: Exception in task 0.0 in stage 704.0 (TID 5010)
java.lang.Exception: sparklyr worker rscript failure with status 255, check worker logs for details.
    at sparklyr.Rscript.init(rscript.scala:83)
    at sparklyr.WorkerApply$$anon$2.run(workerapply.scala:125)
21/06/09 22:48:01 INFO sparklyr: Session (3130) is terminating backend
21/06/09 22:48:01 ERROR TaskSetManager: Task 0 in stage 704.0 failed 1 times; aborting job
...
1 Answers

Linking the GitHub Issue where this problem was resolved.

https://github.com/sparklyr/sparklyr/issues/3139

The options presented by the sparklyr team were one of two options.

  1. You could set the columns argument to columns=c(Species="factor").
  2. You could set the number of records used to infer the schema
config <- spark_config()
config$`sparklyr.apply.schema.infer` <- 150
Related