How to tune hyperparameters of score_func (feature selection) in sklearn?

user_4727203 0
Viewed 545

I'm trying to tune or search for parameters for a scoring function in scikit-learn.

For example, in the pipeline below, I first perform feature selection with SelectKBest, which requires a scoring function (e.g., mutual_info_regression), and finally pass the best features to LinearRegression().

I want to tune the hyperparameter n_neighbors in the mutual_info_regression function, which is the scoring function provided to SelectKBest, but it isn't clear to me how I can tune n_neighbors?

Appreciate any help anyone can provide. Thanks!

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LinearRegression
from sklearn.feature_selection import SelectKBest, mutual_info_regression
from sklearn.model_selection import GridSearchCV

# generate some data
X = np.random.normal(size=(10, 15))
y = np.random.normal(size=10)

# test scoring function
# default hyperparameter is n_neighbors=3
mutual_info_regression(X, y, n_neighbors=3)

# create pipeline
kbest = SelectKBest(mutual_info_regression)  # using default n_neighbors value
pipe = make_pipeline(kbest, LinearRegression())

# how to tune search space for mutual_info_regression n_neighbors?
params = {"selectkbest__score_func": []}  # how to define n_neighbors?
grid = GridSearchCV(pipe, params)
scikit-learn python
1 Answers

Below is a solution I've come up with but I'm not sure if it's the simplest/best way to tune such a hyperparameter.

I've used functools.partial to create partial objects when creating the parameters grid dictionary params, and each object has a different n_neighbors value.

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LinearRegression
from sklearn.feature_selection import SelectKBest, mutual_info_regression
from sklearn.model_selection import GridSearchCV
from functools import partial  # to create partial objects

X = np.random.normal(size=(10, 15))
y = np.random.normal(size=10)
kbest = SelectKBest(mutual_info_regression)
pipe = make_pipeline(kbest, LinearRegression())

# use functools.partial
params = {
    "selectkbest__score_func": [
        partial(mutual_info_regression, n_neighbors=n) for n in range(1, 5)
    ]
}
grid = GridSearchCV(pipe, params)
user_47272030
Related