How to properly GridSearchCV on Bagged DecisionTreeClassifiers

user_17258971 0
Viewed 253

I'm currently trying to hyper-parameter tune through GridSearch an ensemble model composed by DecisionTreeclassifiers bagged with BaggingClassifier. I would like to be able to properly Gridsearch through the following parameters:

Classifier:

'classifier__class_weight': ['balanced'],
'classifier__criterion': ['gini', 'entropy'],
'classifier__max_depth':[1,2,3,4,5,6],
'classifier__max_features':[1,'auto'],
'classifier__min_weight_fraction_leaf': [0.001, 0.005, 0.01],
'classifier__random_state': [0],
'classifier__splitter': ['best','random']

Bagger:

'bagging__bootstrap': [True,False],
'bagging__bootstrap_features': False,
'bagging__max_features': [1.0,2.0,3.0],
'bagging__max_samples': [avgUniqueness,1.0],
'bagging__n_estimators': [10,50,100,1000]

I would like to tune them both simultaneously if possible.

I can not implement the pipeline right away because DecisionTreeClassifier lacks a fit_transform param. But I have the intuition that that's not the correct way.

Maybe I'm not seeing some basic idea. Similar questions are responded with:

param_grid = {
'base_estimator__max_depth' : [1, 2, 3, 4, 5],
'max_samples' : [0.05, 0.1, 0.2, 0.5]
}

clf = GridSearchCV(BaggingClassifier(DecisionTreeClassifier(),
                    n_estimators = 100, max_features = 0.5),
                    param_grid, scoring = choosen_scoring)

But with this solution you can just hyper-tune the classifier rather than the whole ensemble at once. It's also important to mention that I need to pass a fixed sample_weight parameter to the classifier and that "avgUniqueness" is a int value that controls the number of samples for each tree.

cross-validation ensemble-learning hyperparameters scikit-learn python
1 Answers

I've found the solution. When creating the BaggingClassifier:

classifier = BaggingClassifier(base_estimator=None, n_estimators=10, *, max_samples=1.0, max_features=1.0, bootstrap=True, bootstrap_features=False, oob_score=False)

You can pass parameters to base_estimator through "base_estimator__parameter"

Clearly explained in:

Tuning parameters of the classifier used by BaggingClassifier

user_172589710
Related