I am using Spark with Scala. I want to do different preprocessing on my data. Is there a way for CrossValidator to take multiple models (also with ParamMaps) to get the best model out of these two? e.g. What I want to do is:
val discretizer = new QuantileDiscretizer()
.setInputCol("column1")
.setOutputCol("column1disc")
.setNumbuckets(5)
val normalizer = new Normalizer()
.setInputCol("column1")
.setOutputCol("column1norm")
val lr1 = new LinearRegression()
.setFeaturescol(discretizer.getOutputCol)
.setMaxIter(10)
val lr2 = new LinearRegression()
.setFeaturescol(normalizer.getOutputCol)
.setMaxIter(10)
val pipeline = new Pipeline().setStages(Array(dicretizer,normalizer,lr1,lr2))
Now I want my CrossValidator to pick the best of the two models from lr1 and lr2. This is just a small example, I want to extend it to multiple such possibilities with ParamMaps too.