How do I output the regression prediction from each tree in a Random Forest in Python scikit-learn?

user_2596982 0
Viewed 10502

Is there is a way to get the predictions from every tree in a random forest in addition to the combined prediction? I would like to output all of the predictions in a list and not view the entire tree. I know that I can get the leaf indices using the apply method, but I'm not sure how to use that to get the value from the leaf.

Edit: Here's what I have so far from comments below. It wasn't clear to me before that the trees in the estimators_ attribute could be called, but it seems that the predict method can be used on each tree using that attribute. Is this the best way to do this, though?

numberTrees = 100
clf = RandomForestRegressor(n_estimators=numberTrees)
clf.fit(X,Y)
for tree in range(numberTrees):
    print(clf.estimators_[tree].predict(val.irow(1)))
scikit-learn random-forest python
4 Answers

I had the same issue and I don't know how you got the right answer by using print(clf.estimators_[tree].predict(val.irow(1))). It gave me random numbers instead of the actual class. After reading the source code in SKlearn, I realized that we actually have to use predict_proba() instead of predict in the code and it gives you the class that the tree predicts according to the order in clf.classes_. For example:

tree_num = 2
tree_pred = clf.estimators_[tree_num].predict_proba(data_test)
print clf.classes_  #gives you the order of the classes
print tree_pred  #gives you an array of 0 with the predicted class as 1
>>> ['class1','class2','class3']
>>> [0, 1, 0]

You can also use cls.predict_proba() on your data and it gives you the probability of each class prediction by the accumulation of trees and releases you from the pain of going through each tree yourself:

x = clf.predict_proba(data_test) # assume data_test has two instances
print rfc.classes_
print x
>>> ['class1', 'class2', 'class3']
>>> [[0.12 ,  0.02,  0.86], # probabilities for the first instance
     [0.35 ,  0.01,  0.64]]  # for the second instance
user_78242040

What I have done recently is modify sklearn source code to get it. Inside sklearn package sklearn.ensemble.Randomforestregressor

There is a function that if you add print you will see the individual results of each tree. You could change this to be a return and get the individual results of each tree. 

def _accumulate_prediction(predict, X, out, lock):
    """
    This is a utility function for joblib's Parallel.

    It can't go locally in ForestClassifier or ForestRegressor, because joblib
    complains that it cannot pickle it when placed there.
    """
    prediction = predict(X, check_input=False)
    print(prediction)
    with lock:
        if len(out) == 1:
            out[0] += prediction
        else:
            for i in range(len(out)):
                out[i] += prediction[i]

This is a bit more sophisticated since you have to modify the sklearn source code

user_106674120
Related