Cross-validating multiple models using different preprocessing in spark/scala

Viewed 336

I am using Spark with Scala. I want to do different preprocessing on my data. Is there a way for CrossValidator to take multiple models (also with ParamMaps) to get the best model out of these two? e.g. What I want to do is:

val discretizer = new QuantileDiscretizer()
  .setInputCol("column1")
  .setOutputCol("column1disc")
  .setNumbuckets(5)
val normalizer = new Normalizer()
  .setInputCol("column1")
  .setOutputCol("column1norm")

val lr1 = new LinearRegression()
  .setFeaturescol(discretizer.getOutputCol)
  .setMaxIter(10)
val lr2 = new LinearRegression()
  .setFeaturescol(normalizer.getOutputCol)
  .setMaxIter(10)

val pipeline = new Pipeline().setStages(Array(dicretizer,normalizer,lr1,lr2))

Now I want my CrossValidator to pick the best of the two models from lr1 and lr2. This is just a small example, I want to extend it to multiple such possibilities with ParamMaps too.

1 Answers
Related