How to define 'min sample split' and 'min sample leaf' in decision tree regressor?

user_14896942 0
Viewed 1553

I am working on a dataset composed by 20060 rows and 10 columns and I am approaching decision tree regressor to make prediction.

My willing is to use RandomizedsearchCV in order to tune hyperparameters; my doubt is what to write in the dictionary as value for 'min_sample_leaf' and 'min_sample_split'.

My professor told me to rely on the database dimension but I don't understand how!

This is a code example:

def model(model,params):
    r_2 = [] 
    mae_ = []
    rs= RandomizedSearchCV(model,params, cv=5, n_jobs=-1, n_iter=30)
    start = time()
    rs.fit(X_train,y_train)
    #prediction on test data
    y_pred =rs.predict(X_test)
    #R2
    r2= r2_score(y_test, y_pred).round(decimals=2)
    print('R2 on test set: %.2f' %r2)
    r_2.append(r2)
    #MAE
    mae = mean_absolute_error(y_test, y_pred).round(decimals=2)
    print('Mean absolute Error: %.2f' %mae)
    mae_.append(mae)
    #print running time
    print('RandomizedSearchCV took: %.2f' %(time() - start),'seconds')
    return r_2, mae_ 

params= {

    'min_samples_split':np.arange(), #define these two hypeparameter relying on database???
    'min_samples_leaf':np.arange()
}

DT = model(DecisionTreeRegressor(), params)

Can anybody explain me?

Thank you very much

scikit-learn decision-tree python-3.x python
3 Answers

It really depends on what you are trying to achieve. If your goal is to have a model that is usable in production; meaning that it has an acceptable footprint, you may need to increase those values to make sure you have a smaller tree depth mean. Otherwise, all your trees will have too many nodes and your model will take way too much space in memory for inference. Another problem could be the time it will take to train your model. For example, it could take around 1000 seconds to fit an RF with around 4M samples with the dimensions 'n_estimators': 50, 'max_depth': 200, 'min_samples_leaf': 1, 'min_samples_split': 2, 'n_jobs': 20 but only 50s for 'n_estimators': 50, 'max_depth': 200, 'min_samples_leaf': 2, 'min_samples_split': 10, 'n_jobs': 20 for similar accuracy.

If you only care about accuracy, you can set those value fairly low in order to get as much node as possible and learn extensively on your training data. Make sure to do a careful validation (K-fold ?) to avoid overfitting.

Tuning is an Art where you need to balance practical application and optimal metric scores.

user_171428350

What your professor said is to check your data size so that you can decide your parameter values.

For DecisionTreeRegressor, you can see that min_samples_split and min_samples_leaf depend on your n_samples which is the number of rows. The documentation says the same thing for both parameters:

min_samples_split: int or float, default=2

The minimum number of samples required to split an internal node:

· If int, then consider min_samples_split as the minimum number.

· If float, then min_samples_split is a fraction and ceil(min_samples_split * n_samples) are the minimum number of samples for each split.

As the documentation says, if you want to use the parameters making reference to the n_samples (as your teacher says to you), you have to use floats that will represent a fraction (between 0.0 and 1.0) of your number of samples.

For example if you want define min_sample_split is 100, you can write it with two ways: simply 100 or you using the float format 0.005 (you can see that 0.005*20060 is equal to 100).

Using floats allow you to use values that are independent of the number of samples. This is an advantage.

Anyway, I will tell you that probably you are not going to find some big improvements, since the default is super small.

This is applicable for min_sample_split and min_samples_leaf.

user_157840490

Doesn't RandomizedSearchCV already do this for you by default? If refit is true, then your model should be able to give the parameter attributes you are looking for. Something like this:

    rs = RandomizedSearchCV(model, cv=5, n_jobs=-1, n_iter=30)

    search = rs.fit(X_train,y_train)

    search.best_params_

scikit-learn best_index_int The index (of the cv_results_ arrays) which corresponds to the best candidate parameter setting.

The dict at search.cv_results_['params'][search.best_index_] gives the parameter setting for the best model, that gives the highest mean score (search.best_score_).

For multi-metric evaluation, this is not available if refit is False. See refit parameter for more information.

user_132962530
Related