cross validation in pyspark

Viewed 13812

I used cross validation to train a linear regression model using the following code:

from pyspark.ml.evaluation import RegressionEvaluator

lr = LinearRegression(maxIter=maxIteration)
modelEvaluator=RegressionEvaluator()
pipeline = Pipeline(stages=[lr])
paramGrid = ParamGridBuilder().addGrid(lr.regParam, [0.1, 0.01]).addGrid(lr.elasticNetParam, [0, 1]).build()

crossval = CrossValidator(estimator=pipeline,
                          estimatorParamMaps=paramGrid,
                          evaluator=modelEvaluator,
                          numFolds=3)

cvModel = crossval.fit(training)

now I want to draw the roc curve, I used the following code but I get this error:

'LinearRegressionTrainingSummary' object has no attribute 'areaUnderROC'

trainingSummary = cvModel.bestModel.stages[-1].summary
trainingSummary.roc.show()
print("areaUnderROC: " + str(trainingSummary.areaUnderROC))

I also want to check the objectiveHistory at each itaration, I know that I can get it at the end

print("numIterations: %d" % trainingSummary.totalIterations)
print("objectiveHistory: %s" % str(trainingSummary.objectiveHistory))

but I want to get it at each iteration, how can I do this?

Moreover I want to evaluate the model on the test data, how can I do that?

prediction = cvModel.transform(test)

I know for the training data set I can write:

print("RMSE: %f" % trainingSummary.rootMeanSquaredError)
print("r2: %f" % trainingSummary.r2)

but how can I get these metrics for testing data set?

1 Answers
Related