Difference between AUPRC in caret and PRROC

Question

I'm working in a very unbalanced classification problem, and I'm using AUPRC as metric in caret. I'm getting very differents results for the test set in AUPRC from caret and in AUPRC from package PRROC.

In order to make it easy, the reproducible example uses PimaIndiansDiabetes dataset from package mlbench:

rm(list=ls())
library(caret)
library(mlbench)
library(PRROC)

#load data, renaming it to 'datos'
data(PimaIndiansDiabetes)
datos=PimaIndiansDiabetes[,1:9]

# training and test
set.seed(998)
inTraining <- createDataPartition(datos[,9], p = .8, list = FALSE)
training <-datos[ inTraining,]
testing <- datos[ -inTraining,]

#training

 control=trainControl(method = "cv",summaryFunction = prSummary,
 classProbs = TRUE)
 set.seed(998)
 rf.tune <-train(training[,1:8],training[,9],method ="rf",   
 trControl=control,metric="AUC")

#evaluating AUPRC in test set

 matriz=cbind(testing[,9],predict(rf.tune,testing[,1:8],type="prob"),
 predict(rf.tune,testing[,1:8]))
 names(matriz)=c("obs",levels(testing[,9]),"pred")
 prSummary(matriz,levels(testing[,9]))


 #calculating AUPRC through pr.curve

#checking positive class
 confusionMatrix(predict(rf.tune,testing[,1:8]),testing[,9],
 mode  = "prec_recall")#'Positive' Class : neg 

#preparing data for pr.curve
indice_POS=which(testing[,9]=="neg")
indice_NEG=which(testing[,9]=="pos")

#the classification scores of  only the data points belonging to the 
#positive class
 clas_score_POS=predict(rf.tune,testing[,1:8],type="prob")[indice_POS,1]

 #the classification scores of  only the data points belonging to the 
 #negative class
 clas_score_NEG=predict(rf.tune,testing[,1:8],type="prob")[indice_NEG,2]

 pr.curve(clas_score_POS,clas_score_NEG)

Value from PRROC is 0.9053432 and from caret prSummary is 0.8714607. In my unbalanced case, the differences are broader(AUPRC= 0.1688446 with SMOTE resampling -via control$sampling <- "smote"- and 0.01429 with PRROC.)

Is this because of the different methods to calculate AUPRC in those packages or I'm doing something wrong?

UPDATED: I can't find bugs in my code. After missuse answer, I'd like to make some remarks:

When you do prSummary(matriz,levels(testing[,9])) you got

 AUC      Precision    Recall         F 
0.8714607 0.7894737 0.9000000 0.8411215

which is consistent with

confusionMatrix(predict(rf.tune,testing[,1:8]),testing[,9],mode  = "prec_recall")
Confusion Matrix and Statistics

          Reference
Prediction neg pos
       neg  90  23
       pos  10  30

               Accuracy : 0.7843          
                 95% CI : (0.7106, 0.8466)
    No Information Rate : 0.6536          
    P-Value [Acc > NIR] : 0.0003018       

                  Kappa : 0.4945          
 Mcnemar's Test P-Value : 0.0367139       

              Precision : 0.7965          
                 Recall : 0.9000          
                     F1 : 0.8451          
             Prevalence : 0.6536          
         Detection Rate : 0.5882          
   Detection Prevalence : 0.7386          
      Balanced Accuracy : 0.7330          

       'Positive' Class : neg

And with:

> MLmetrics::PRAUC(y_pred = matriz$neg, y_true = ifelse(matriz$obs == "neg", 1, 0))
[1] 0.8714607

As you can see in the last line, the 'Positive' class is 'neg', and I think that missuse is considering the positive class as 'pos', so we have different metrics. Moreover, when you print the trained rf, the results are also consistent with an expected AUC~0.87:

> rf.tune
Random Forest 

615 samples
  8 predictor
  2 classes: 'neg', 'pos' 

No pre-processing
Resampling: Cross-Validated (10 fold) 
Summary of sample sizes: 554, 553, 553, 554, 554, 554, ... 
Resampling results across tuning parameters:

  mtry  AUC        Precision  Recall  F        
  2     0.8794965  0.7958683  0.8525  0.8214760
  5     0.8786427  0.8048463  0.8325  0.8163032
  8     0.8528028  0.8110820  0.8325  0.8192225

I'm not worried about the difference 0.87caret-0.9PRROC in this case, but I'm very worried about 0.1688446 caret/ 0.01429 PRROC in the unbalanced case. Might this be because the numeric divergence under different implementations is strengthened in the unbalanced case? And if there are a numerical difference in the implementations, how's that they are identical 0.8714607 in the test set?

user_8573068 · Answer

I trust you are making several mistakes in you code.

First of all caret::prSummary uses MLmetrics::PRAUC to compute the AUPRC. It should be defined like this:

MLmetrics::PRAUC(y_pred = matriz$pos, y_true = ifelse(matriz$obs == "pos", 1, 0))

#output
0.7066323

using the positive class probability and the numeric 0/1 vector of true classes (1 for positive)

The same result is obtained by using:

caret::prSummary(matriz, levels(testing[,9])[2])

MLmetrics::PRAUC uses ROCR::prediction to construct the curve:

pred_obj <- ROCR::prediction(matriz$pos, ifelse(matriz$obs == "pos", 1, 0))
perf_obj <- ROCR::performance(pred_obj, measure = "prec", 
                              x.measure = "rec")

and the curve looks like:

ROCR::plot(perf_obj, ylim = c(0,1))

when one uses PRROC::pr.curve there are several ways to define the inputs. One is to provide a vector of probabilities for the positive class for the positive observations, and a vector of probabilities for the positive class for the negative observations:

preds <- predict(rf.tune,
                 testing[,1:8],
                 type="prob")[,2] #prob of positive class
preds_pos <- preds[testing[,9]=="pos"] #preds for true positive class
preds_neg <- preds[testing[,9]=="neg"] #preds for true negative class

PRROC::pr.curve(preds_pos, preds_neg)
#truncated output
0.7254904

these two numbers (obtained by PRROC::pr.curve and MLmetrics::PRAUC) do not match

however the curve

plot(PRROC::pr.curve(preds_pos, preds_neg, curve = TRUE))

looks just like the above one obtained using ROCR::plot.

To check:

res <- PRROC::pr.curve(preds_pos, preds_neg, curve = TRUE)

ROCR::plot(perf_obj, ylim = c(0,1), lty = 2, lwd = 2)
lines(res$curve[,1], res$curve[,2], col = "red", lty = 5)

they are the same. Therefore the difference in the obtained area is due to different implementations in the mentioned packages.

These implementations can be checked by looking at the source for:

MLmetrics:::Area_Under_Curve #this one looks pretty straight forward 
PRROC:::compute.pr #haven't the time to study this one but if I had to bet I'd say this one is more accurate for step like curves.

Difference between AUPRC in caret and PRROC

1 Answers