Fitting customized LGBM parameters in sklearn pipeline

user_10360685 0
Viewed 672

I am working on a binary classifier using LightGBM. My classifier definition looks like following:

# sklearn version, for the sake of calibration
bst_ = LGBMClassifier(**search_params, **static_params, n_estimators = 1500)

bst_.fit(X = X_train, y = y_train, sample_weight = TRAIN_WEIGHTS,
         eval_set = (X_test, y_test), eval_sample_weight = [TEST_WEIGHTS],
         eval_metric = my_scorer,
         early_stopping_rounds = 150, 
         callbacks = [lgb.reset_parameter(learning_rate = lambda current_round: learning_rate_decay(current_round, 
                                                                                                    base_learning_rate = learning_rate,
                                                                                                    decay_power = decay_power))],
         categorical_feature = cat_vars)

where **search_params are hyperparameters optimized by Optuna and **static_params are predefined parameters such as 'objective' or 'random_state'.

On top of that I define weights on each of the target using sample_weight, I use customized objective function my_scorer, early stopping and decaying learning rate defined as below:

def learning_rate_decay(current_iter, base_learning_rate = 0.05, decay_power = 0.99):
    lr = base_learning_rate  * np.power(decay_power, current_iter)
    return lr if lr > 1e-3 else 1e-3

As I want to have probabilities as a result of my modelling, I would like to use isotonic regression as a final part of the forecasting pipeline. I know that I could use following code:

# Calibrate 
calibrated_clf = CalibratedClassifierCV(
    base_estimator=bst_,
    method = 'isotonic',
    cv="prefit"
)
calibrated_clf.fit(X_train, y_train)

but according to this, I shouldn't be using "prefit" on train dataset. I would like to create a pipeline, which would maintain all the arguments defined in my classifier .fit (such as callback), something like this:

calibrated_clf = CalibratedClassifierCV(
    base_estimator=bst_,
    method='isotonic',
    cv=5
)
calibrated_clf.fit(X_train, y_train)
calibrated_clf.set_params(**customized_lgbm_params)

yet obviously it doesn't work, as those are specific parameters for fitting the data to the model.

My question is: how could I define a pipeline which would include all the features of LGBMClassifier.fit that I have already defined (such as early stopping, applying weights, callbacks)?

lightgbm scikit-learn classification machine-learning python
1 Answers

Use sklearn's pipeline

pipeline = Pipeline([
    ("classifier", CalibratedClassifier(
        base_estimator=bst_(**search_params),
        method='isotonic', 
        cv=5) 
    )
])
user_73024040
Related