scikit-learn: most important feature due to SelectKBest() is not the same feature of top node in DecisionTreeClassifier() with unedited data?

user_12778634 0
Viewed 313

I am applying the breast cancer dataset to a decision tree as simple as possible:

from sklearn.datasets import load_breast_cancer
from sklearn.feature_selection import SelectKBest
from sklearn.model_selection import train_test_split
import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sklearn.tree import export_graphviz
import graphviz

cancer = load_breast_cancer()

#print(cancer.feature_names)

X_train, X_test, y_train, y_test = train_test_split(cancer.data, cancer.target, random_state=0)

tree = DecisionTreeClassifier(random_state=0, max_depth=2)
tree.fit(X_train, y_train)

print(f"\nscore train: {tree.score(X_train, y_train)}")
print(f"score test : {tree.score(X_test, y_test)}")
>>>
score train: 0.9413145539906104
score test : 0.9370629370629371

export_graphviz(tree, out_file=f"./src/dot/testing/breast_cancer.dot", class_names=['malignant', 'benign'], feature_names=cancer.feature_names, impurity=False, filled=True)

with open(f"./src/dot/testing/breast_cancer.dot") as f:
    dot_graph = f.read()
graphviz.Source(dot_graph)

Which lead to this graph: enter image description here

Playing with feature selection, I want to get only the most important feature. In my understanding it should be the feature in the root-leaf, no? Unfortunately it's not, it's "worst concave points". Here is what I did to get the most important feature:

select = SelectKBest(k=1)
select.fit(X_train, y_train)

X_train_selected = select.transform(X_train)

print("X_train.shape         : {}".format(X_train.shape))
print("X_train_selected.shape: {}\n".format(X_train_selected.shape))
>>>
X_train.shape         : (426, 30)
X_train_selected.shape: (426, 1)

mask = select.get_support()

# plt.matshow(mask.reshape(1, -1), cmap='gray_r')
# plt.xlabel("Sample index")

print("most important features:")
for mask, feature in zip(mask, cancer.feature_names):
    if mask: print(feature)
>>>
most important features:
worst concave points

I guess I am getting something wrong here. Could somebody clarify this? Any hint? Thanks

scikit-learn python-3.x
1 Answers

The most important feature does not necessarily mean that it will be the one used to make the first split. In fact, sklearn.tree.DecisionTreeClassifier uses entropy to decide which feature to use when making a split, so unless SelectKBest does this too, there is no need for both methods to reach the same conclusions in the same order. Even the same feature will reduce entropy differently in different stages of a tree classifier.

As a side note, trees do not always consider all features when making nodes. Take a look at max_features here. This means that, depending on your random-state and max_features hyper parameters, your tree may or may not have considered worst_concave_points when making the first split.

user_97958170
Related