scikit learn kernel ridge regression produces very different result using linear kernel than simple linear regression

W Jin 0
Viewed 38

My understanding of the kernel regression is when using linear kernel for ridge regression with no penalty, results should be similar to linear regression. somehow in the toy example linear regression has much better Rsq. part of the problem I think is SK is using sample space for the kernel matrix instead of the smaller of sample and feature space and in this case feature space is much smaller. how do I "fix" the issue or maybe I have done something wrong - a newbie to kernel regression. linear kernel regression has around -3 score and linear regression score is about 50%

from sklearn.datasets import load_diabetes
from sklearn.kernel_ridge import KernelRidge
from sklearn.utils import shuffle
from sklearn.metrics.pairwise import pairwise_kernels
import numpy as np


diab = load_diabetes()
X, y = shuffle(diab.data, diab.target)
X_train, X_test = X[:1000, :], X[100:, :]
y_train, y_test = y[:1000], y[100:]

# krr = KernelRidge(kernel='polynomial', degree=2, coef0=1)
# krr = KernelRidge(kernel='rbf', degree=2, coef0=0)
krr = KernelRidge(kernel='linear', alpha=0.0001)
# krr = KernelRidge(kernel='laplacian')

kernel_train = np.dot(X_train, X_train.T)  # linear kernel

# krr.fit(kernel_train, y_train)
krr.fit(X_train, y_train)

#kernel_test = np.dot(X_test, X_train[svc.support_, :].T)

# kernel_test = np.dot(X_test, X_train.T)
# y_pred = krr.predict(kernel_test)
y_pred = krr.predict(X_test)
# krr.score(kernel_test, y_test), krr.score(kernel_train, y_train)
krr.score(X_test, y_test)
regression scikit-learn python
1 Answers

As far as I know, sklearn KernelRidge() implementation does not support intercept. Try LinearRegression(fit_intercept=False) and you should get similarly poor R2. Plot your targets against predictions and you'll see it actually predicts OK, except the prediction is zero centered.

What can you do?

Easy approximation: preprocess your target to zero center it, or estimate the intercept from the mean target values.

Hard way: KernelCenterer():

from sklearn.preprocessing import KernelCenterer
from sklearn.metrics import r2_score

kernel_train = X_train @ X_train.T # That's what linear kernel does.
centerer = KernelCenterer()
kernel_train_centered = centerer.fit_transform(kernel_train)

clf = KernelRidge(alpha=0.0001, kernel="precomputed")
clf.fit(kernel_train_centered, y_train - y_train.mean())

kernel_test = X_test @ X_train.T
y_pred = clf.predict(kernel_test)

intercept = y_train.mean() - centerer.K_fit_rows_ @ clf.dual_coef_

y_pred += intercept

r2_score(y_test, y_pred)
dx2-660
Related