how to perform statistical significant test to compare output values in python

user_9917763 0
Viewed 78

basically I have this dataframe:

baseline_data: userid|y_pred
                 12  |29
                 13  |35
                 14  |32 
                 15  |39

and I have created a model and got the age data: df

userid|age
12    |31
13    |37
14    |32
15    |36

how can I run a statistical significance test to check that this model performs significantly better than the baseline prediction ?

pandas dataframe python
1 Answers

Without validation dataset how can one conduct statistical analysis? For simulation/prediction models you need two sets of sample/observed data. One set will be used to train the model and another set will be used to validate the model predictions. Generally, the training and validation data set is split into an 80:20 ratio. So, you need to select your validation dataset size based on training dataset size.

Let's assume you have trained your model based on a training dataset of d1 and you have another validation dataset d2 for the userids that you have predicted using your models.

d2:

   userid  age
0      12   30
1      13   36
2      14   32
3      15   37

Now, you want to compare d2.age with your previous two models data. Let's assume other two dfs are m1 and m2 respectively. they are m1.y_pred and m2.age.

import numpy as np
rmse_m1=np.sqrt(((m1.y_pred - d2.age) ** 2).mean())
rmse_m2=np.sqrt(((m2.age - d2.age) ** 2).mean())
rmse_m1:
1.224744871391589
rmse_m2:
0.8660254037844386

Now, lower values of RMSE indicate better fit. So, does your m2 model predictions are better? Well, though the second value of RMSE is smaller than the other; you can check if the difference is statistically significant to be able to say that one model is better than the other.

For more information read Confidence interval of RMSE

user_111955640
Related