How to get the prediction probabilities using cross validation in scikit-learn

user_10704050 0
Viewed 1321

I am using RandomForestClassifier as follows using cross validation for a binary classification (class labels are 0 and 1).

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score

clf=RandomForestClassifier(random_state = 42, class_weight="balanced")
k_fold = StratifiedKFold(n_splits=10, shuffle=True, random_state=0)
accuracy = cross_val_score(clf, X, y, cv=k_fold, scoring = 'accuracy')
print("Accuracy: " + str(round(100*accuracy.mean(), 2)) + "%")
f1 = cross_val_score(clf, X, y, cv=k_fold, scoring = 'f1_weighted')
print("F Measure: " + str(round(100*f1.mean(), 2)) + "%")

Now I want to order my data using prediction probabilities of class 1 with cross validation results. For that I tried the following two ways.

pred = clf.predict_proba(X)[:,1]
print(pred)

probs = clf.predict_proba(X)
best_n = np.argsort(probs, axis=1)[:,-6:]

I get the following error

NotFittedError: This RandomForestClassifier instance is not fitted yet. Call 'fit' with appropriate arguments before using this method.

for both the situations.

I am just wondering where I am making things wrong.

I am happy to provide more details if needed.

cross-validation scikit-learn random-forest python
3 Answers

In case, you want to use the CV model for a unseen data point/s, use the following approach.

from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_validate

iris = datasets.load_iris()
X = iris.data
y = iris.target
clf = RandomForestClassifier(n_estimators=10, random_state = 42, class_weight="balanced")

cv_results = cross_validate(clf, X, y, cv=3, return_estimator=True)

clf_fold_0 = cv_results['estimator'][0]

clf_fold_0.predict_proba([iris.data[133]])

# array([[0. , 0.5, 0.5]])
user_63476290

Have a look at the documentation it specifies that the probability is calculated based on the mean results from the trees.

In your case, you first need to call the fit() method to generate the tress in the model. Once you fit the model on the training data, you can call the predict_proba() method.

This is also specified in the error.

# Fit model
model = RandomForestClassifier(...)
model.fit(X_train, Y_train)

# Probabilty
model.predict_proba(X)[:,1]
user_115234000

I solved my problem using the following code:

proba = cross_val_predict(clf, X, y, cv=k_fold, method='predict_proba')
print(proba[:,1])
print(np.argsort(proba[:,1]))
user_107040500
Related