My aim is to add a rank based evaluator to the CrossValidator function (PySpark)
cvExplicit = CrossValidator(estimator=cvSet, numFolds=8, estimatorParamMaps=paramMap,evaluator=rnkEvaluate)
Although I need to pass the evaluated dataframe into the function, and I do not know how to do that part.
class rnkEvaluate():
def __init__(self, user_col = "user", rating_col ="rating", prediction_col = "prediction"):
print(user_col)
print(rating_col)
print(prediction_col)
def isLargerBetter():
return True
def evaluate(self,predictions):
denominator =
predictions.groupBy().sum(self._rating_col).collect()[0][0]
TODO
rest of the calculation ...
return numerator / denominator
Somehow I need to pass the predictions dataframe at every fold iteration, but I could not manage it.