What's the best way to Create the Confusion matrix and thereby the evaluation parameters like Accuracy, Precision and recall in Pyspark 3.0+. I have seen others answers but they are too slow for even a small pyspark dataframe with just 800K rows with the label and predictions.
The solution here Confusion Matrix to get precsion,recall, f1score does a sorting of the rows which is very slow if the dataframe is huge. Is ther no other better way. I also tried the collect function ( also suggested in the same link) but thats slow aswell.