Custom Evaluator during cross validation SPARK

Viewed 1245

My aim is to add a rank based evaluator to the CrossValidator function (PySpark)

cvExplicit = CrossValidator(estimator=cvSet, numFolds=8, estimatorParamMaps=paramMap,evaluator=rnkEvaluate)

Although I need to pass the evaluated dataframe into the function, and I do not know how to do that part.

class rnkEvaluate():
def __init__(self, user_col = "user", rating_col ="rating", prediction_col = "prediction"):
    print(user_col)
    print(rating_col)
    print(prediction_col)

def isLargerBetter():
    return True


def evaluate(self,predictions):
    denominator = 
    predictions.groupBy().sum(self._rating_col).collect()[0][0]
    TODO 
    rest of the calculation ...
    return numerator / denominator

Somehow I need to pass the predictions dataframe at every fold iteration, but I could not manage it.

1 Answers
Related