“UserWarning: One or more of the test scores are non-finite” warning only when adding RandomForest max_features parameter to RandomizedSearchCV

Viewed 3443
from sklearn.model_selection import RandomizedSearchCV

# --initialise classifier
classifier = RandomForestClassifier(n_estimators=300)

# -- set hyperparameters to tune
param_grid = {
   "max_depth": np.arange(20, 60, 10),
   "min_samples_leaf": np.arange(1, 15),
   'max_features': np.arange(0, 1, 0.05),
}

random = np.random.RandomState(42)

# -- initialise grid search
random_model_search = RandomizedSearchCV(
    estimator=classifier,
    param_distributions=param_grid,
    n_iter=100,
    scoring="f1",
    return_train_score=True,
    n_jobs=-1,
    cv=3,
    random_state=random
)

# -- fit the model and extract best score
random_model_search.fit(X_train_encoded, Y_train)
print(f"Best score: {random_model_search.best_score_}")

print("Best parameters set:")
best_parameters_random = random_model_search.best_estimator_.get_params()
for param_name in sorted(param_grid.keys()):
    print(f"\t{param_name}: {best_parameters_random[param_name]}")

When I run this code with max_depth in param_grid, I get a UserWarning saying some test scores are nan values. However, if I take this hyperparameter out, random search runs just fine, no warning. I understand that this warning comes up when there's categories in the validation/test set that are not present in the training set and so have not been properly encoded. I am using the train set for this random search with the whole of the train set encoded, so I'm not sure why this warning comes up? Can anyone advise on this?

The encoding and scaling code is below:

# Set encoding and scaling instructions
column_transform = make_column_transformer(
    (OneHotEncoder(handle_unknown = "ignore"), columns_for_onehot),
    (OrdinalEncoder(categories=[ordinal_order], handle_unknown='use_encoded_value', unknown_value=3), columns_for_ordinal),
    remainder=MinMaxScaler()
)

# Apply column transformer to features
X_train_encoded = column_transform.fit_transform(X_train)
1 Answers

Generally to debug, you should check random_model_search.cv_results_ to find out which hyperparameter combinations lead to nan scores, and whether they occur in all the folds for a given hyperparameter combination.

In this case, I strongly suspect the issue is that max_features=0 is a possibility, and the model will fail to train in that case.

Related