Stacking with custom classifier

user_11282402 0
Viewed 180

I want to build a stacking classifier for the three label following dataset:

from sklearn.datasets import make_classification

# Define dataset
def get_dataset():
    X, y = make_classification(
        n_samples=1000,
        n_features=20,
        n_informative=15,
        n_redundant=5,
        random_state=2022,
        n_classes=3,
    )
    return X, y

I want to use sklearn models plus a dummy model of my own:

from sklearn.base import BaseEstimator, ClassifierMixin
from sklearn.utils.validation import check_X_y, check_array, check_is_fitted
from sklearn.utils.multiclass import unique_labels


class DummyClassifier(BaseEstimator, ClassifierMixin):
    def __init__(self):
        pass

    def fit(self, X, y, **kwargs):
        if y is None:
            raise ValueError("requires y to be passed, but the target y is None")

        X, y = check_X_y(X, y)

        self.n_features_in_ = X.shape[1]
        self.classes_ = np.unique(y)
        self.is_fitted_ = True

        self.X_ = X
        self.y_ = y

        return self

    def predict(self, X):
        check_is_fitted(self, ["is_fitted_", "X_", "y_"])
        X = check_array(X)

        return self.y_[np.ones(X.shape[0], dtype=np.int64)]

    def is_classifier():
        return True

combined:

from sklearn.ensemble import StackingClassifier

def get_stacking():
    # Base models
    level0 = list()
    level0.append(("logistic reg", LogisticRegression()))
    level0.append(("knn", KNeighborsClassifier()))
    level0.append(("cart", DecisionTreeClassifier()))
    level0.append(("svm", SVC()))
    level0.append(("random", DummyClassifier()))
    # Meta learner
    level1 = LogisticRegression()
    # Stacking ensemble
    model = StackingClassifier(estimators=level0, final_estimator=level1, cv=5)
    return model

I get interpretability through the final_estimator's coefficients:

stack = get_stacking()
stack.fit(X, y)
for i in range(3):
    print(stack.final_estimator_.classes_[i])
    print(stack.final_estimator_.coef_[i])

I understand that for 3 classes and 4 models, I get 12 coefficients for each final output class. In my case I have 5 models but 13 coefficients:

0
[ 0.12169113 -0.00882275 -0.10934588  1.39370271 -0.647395   -0.7427852
  0.28413927  0.05880896 -0.33942572  0.17350847 -0.71793331 -0.62128377
 -0.01146239]
1
[ 0.05118664  0.07307435 -0.12623659 -0.7132075   1.22529472 -0.51406282
 -0.16767687 -0.00635399  0.17205527 -0.27319447  0.6866465  -0.47218785
  0.09065114]
2
[-0.17287777 -0.06425161  0.23558246 -0.68049521 -0.57789973  1.25684802
 -0.1164624  -0.05245497  0.16737045  0.099686    0.03128681  1.09347161
 -0.07918875]

Where's the problem with my DummyClassifier?

ensemble-learning scikit-learn python
2 Answers

From the documentation of StackingClassifier, the parameter stack_method is set to 'auto' by default, which will try to call predict_proba, decision_function, and predict in that order. The first three classifiers in the ensemble (LogisticRegression, KNeighborsClassifier, and DecisionTreeClassifier) have a predict_proba method defined, which returns an array of shape (n, c) where n are the number of rows from the input array and c are the number of classes. For example, predicting with the LogisticRegression model (the first model in the list of estimators):

stack.estimators_[0].predict_proba(np.ones((5, 20)))

Will return an array of shape (5, 3) since there are 5 rows in the input array and 3 classes in your example.

The fourth classifier in the ensemble, SVC does not define predict_proba but it does define decision_function, the next in the precedence in order, which also returns an array of shape (n, c).

The DummyClassifier you have defined only defines predict, the last in the precedence order, and it returns an array of shape (1,). When this is concatenated with the other outputs, there will be 4 x 3 + 1 = 13 coefficients, as you're observing. To have 5 x 3 = 15 coefficients, corresponding to 3 outputs per model, the DummyClassifier needs to either change the output of predict, or define a predict_proba or decision_function method. I would recommend doing the latter for consistency, as the predict method in the other models returns predictions in the same format. A simple way to do that would be defining predict_proba which just one-hot encodes the output from the predict method:

    def predict_proba(self, X):
        prediction = self.predict(X)
        output = np.zeros((X.shape[0], self.classes_.size))
        output[np.arange(X.shape[0]), prediction] = 1
        return output

After defining this method in the DummyClassifier, there are 15 output coefficients as expected:

0
[ 0.09247372 -0.01868337 -0.07385955  1.42720504 -0.64726132 -0.78001291
  0.32688202  0.00670499 -0.3336562   0.15328638 -0.7135016  -0.6332385
 -0.00235729  0.          0.0022881 ]
1
[ 0.02883837  0.0887803  -0.11983384 -0.74669545  1.23962866 -0.49514838
 -0.07969914 -0.0489527   0.12643667 -0.26378726  0.69218343 -0.46387066
 -0.09017733  0.          0.08796216]
2
[-0.12131209 -0.07009693  0.19369339 -0.6805096  -0.59236733  1.27516129
 -0.24718288  0.04224771  0.20721954  0.11050088  0.02131817  1.09710915
  0.09253462  0.         -0.09025026]
user_130736770

Your dummy classifier is returning integer-encoded values rather than one-hot encoded values, so it is only adding a single dimension to your final estimator's input.

user_136315910
Related