Scikit learn: RidgeCV seems not to give the best option?

user_6435921 0
Viewed 3395

This is my X:

 X =  np.array([[  5.,   8.,   3.,   4.,   0.,   5.,   4.,   0.,   2.,   5.,  11.,
              3.,  19.,   2.],
           [  5.,   8.,   3.,   4.,   0.,   1.,   4.,   0.,   3.,   5.,  13.,
              4.,  19.,   2.],
           [  5.,   8.,   3.,   4.,   0.,   4.,   4.,   0.,   3.,   5.,  12.,
              2.,  19.,   2.],
           [  5.,   8.,   3.,   4.,   0.,   1.,   4.,   0.,   4.,   5.,  12.,
              4.,  19.,   2.],
           [  5.,   8.,   3.,   4.,   0.,   1.,   4.,   0.,   3.,   5.,  12.,
              5.,  19.,   2.],
           [  5.,   8.,   3.,   4.,   0.,   2.,   4.,   0.,   3.,   5.,  13.,
              3.,  19.,   2.],
           [  5.,   8.,   3.,   4.,   0.,   2.,   4.,   0.,   4.,   5.,  11.,
              4.,  19.,   2.],
           [  5.,   8.,   3.,   4.,   0.,   2.,   4.,   0.,   3.,   5.,  11.,
              5.,  19.,   2.],
           [  5.,   8.,   3.,   4.,   0.,   1.,   4.,   0.,   3.,   5.,  12.,
              5.,  19.,   2.],
           [  5.,   8.,   3.,   4.,   0.,   1.,   4.,   0.,   3.,   5.,  12.,
              5.,  19.,   2.]])

and this is my response y

y = np.array([ 70.14963195,  70.20937046,  70.20890363,  70.14310389,
        70.18076206,  70.13179977,  70.13536797,  70.10700998,
        70.09194074,  70.09958111])

Ridge Regression

    # alpha = 0.1
    model = Ridge(alpha = 0.1)
    model.fit(X,y)
    model.score(X,y)   # gives 0.36898424479816627

    # alpha = 0.01
    model1 = Ridge(alpha = 0.01)
    model1.fit(X,y)
    model1.score(X,y)     # gives 0.3690347045143918 > 0.36898424479816627

    # alpha = 0.001
    model2 = Ridge(alpha = 0.001)
    model2.fit(X,y)
    model2.score(X,y)  #gives 0.36903522192901728 > 0.3690347045143918

    # alpha = 0.0001
    model3 = Ridge(alpha = 0.0001)
    model3.fit(X,y)
    model3.score(X,y)  # gives 0.36903522711624259 > 0.36903522192901728

Thus from here it should be clear that alpha = 0.0001 is the best option. Indeed reading the documentation it says that the score is the coefficient of determination. If the coefficient closest to 1 describes the best model. Now let's see what RidgeCV tells us

RidgeCV regression

modelCV = RidgeCV(alphas = [0.1, 0.01, 0.001,0.0001], store_cv_values = True)
modelCV.fit(X,y)
modelCV.alpha_  #giving 0.1
modelCV.score(X,y)  # giving 0.36898424479812919 which is the same score as ridge regression with alpha = 0.1

What is going wrong? Surely we can check manually, as I have done, that all the other alphas are better. So not only it is not choosing the best alpha, but it is choosing the worst!

Can someone explain to me what it's going wrong?

cross-validation scikit-learn regression statistics python
2 Answers

sascha's answer is correct. Here is a proof that RidgeCV does pick the right alpha.

I write a function to test whether the index of the minimum cross validated error for the alphas match the index of 0.1 in the list of alphas. 

def test_RidgeCV(alphas):
    modelCV = RidgeCV(alphas = alphas, store_cv_values = True)
    modelCV.fit(X,y)
    modelCV.score(X,y)
    # print(modelCV.alpha_)
    CV_values =modelCV.cv_values_
    mean_error = np.min(CV_values, axis=0)
    return alphas.index(0.1) == np.argmin(mean_error)

Then I run through the full permutation of the list of alphas provided in the question. No matter where we put the 0.1, its index always matches the index of the minimum error.
This is exhaustive test. We got 24 True's. 

alphas=[0.1, 0.01, 0.001,0.0001]

from itertools import permutations
alphas_list = list(permutations(alphas))
for i in range(len(alphas_list)):
    print(test_RidgeCV(alphas=alphas_list[i]))

Out: True True ... True

user_90806460
Related