Using StackingClassifier with train/test split instead of CV

user_5559925 0
Viewed 686

I've been experimenting with StackingClassifiers lately, and usually it's used with cross-validation (default: K-fold, num-folds = 5). That's written like so:

from sklearn.pipeline import Pipeline
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, RandomizedSearchCV, KFold

X, y = load_breast_cancer(return_X_y=True, as_frame=True)

model = StackingClassifier(estimators=[
        ('tree', Pipeline([('tree', DecisionTreeClassifier(random_state=42))])),
        ('knn', Pipeline([('knn', KNeighborsClassifier())])),
    ],final_estimator = final_estimator, n_jobs = 10, passthrough = False, cv = KFold())

model.fit(X, y)

which works just fine. However, k-fold cross validation as a way to train a stacking classifier is really quite slow. Ideally i'd like to use to approach where 80% of the training data is used to train the component models, and the other 20% used to train the final_estimator. According to the docs, you can use an iterator that yields a train-test split as an input to cv (Cross Validation) (which is wonky because it's no longer CV). So I suppose two part question

  • Is it possible to use StackingClassifier to work in this way, where it's not using cross-validation(CV) at all but a train/test split (mostly to speed up fitting)?
  • If so, how would this look? what would i set as my input to cv?

One other note. Looking through potential CV options in the user guide ShuffleSplit with nsplits=1 seemed like a good option, and i tried it.

model = StackingClassifier(estimators=[
        ('tree', Pipeline([('tree', DecisionTreeClassifier(random_state=42))])),
        ('knn', Pipeline([('knn', KNeighborsClassifier())])),
    ],final_estimator = final_estimator, n_jobs = 10, passthrough = False, cv = ShuffleSplit(n_splits=1))

model.fit(X, y)

But unfortunately that doesnt work and throws an error:

ValueError: cross_val_predict only works for partitions
ensemble-learning scikit-learn machine-learning python
2 Answers

From the Stacked Classifier page of sklearn, it states that "... estimators_ are fitted on the full X while final_estimator_ is trained using cross-validated predictions of the base estimators using cross_val_predict."

I think if you would like to use 80%-20% based traditional trainig-testing approach, you should do that outside of the StackedClassifier. The primary reason is to keep the process consistent with how would want to use Stacked Classifier per the documented note above.

I have done this and my pipeline / setup descriptively looked as follows. I did not mind the extra time in doing the training-testing with each classifier since the StackedClassifier was processed more efficiently for the final classifier:

  1. Run individual classifiers (KNN, Decision Tree, Random Forests, Naive Bayes etc) with 70%-30% / 80%-20% split. Find the best parametrization.
  2. Setup StackedClassifier where each classifier is fit with the best parameters that you identified with whole data (no splits at this stage)
  3. Check and validate the results of the final classifier relative to the performance of individual classifiers.

I think the extra time taken is unavoidable, since you are introducing an extra step of splitting data for your individual classifiers. For your final classifier CV stage, even if you do a split, the final model input should be run on whole data for maximum efficiency for the individual and final clssifiers.

user_109999650

The fastest speedup you can get is with KFold(n_splits=2):

model = StackingClassifier(estimators=[
        ('tree', Pipeline([('tree', DecisionTreeClassifier(random_state=42))])),
        ('knn', Pipeline([('knn', KNeighborsClassifier())])),
    ],final_estimator = final_estimator, n_jobs = 10, passthrough = False, cv = KFold(n_splits=2))

cv param only accepts "partitions" or "partitioners" ("by defintion" as they say). They are KFold(), StratifiedKFold, but not ShuffleSplit or train_test_split.

Partitioners:

n =5
x = range(90,100)
cv = KFold(n_splits=n).split(x)

for i,j in cv:
    print("TRAIN:",i,"TEST",j)
TRAIN: [2 3 4 5 6 7 8 9] TEST [0 1]
TRAIN: [0 1 4 5 6 7 8 9] TEST [2 3]
TRAIN: [0 1 2 3 6 7 8 9] TEST [4 5]
TRAIN: [0 1 2 3 4 5 8 9] TEST [6 7]
TRAIN: [0 1 2 3 4 5 6 7] TEST [8 9]

Non-partitioners:

n =5
x = range(90,100)
# cv = KFold(n_splits=n).split(x)
cv = ShuffleSplit(n_splits=n, train_size=.8).split(x)

for i,j in cv:
    print("TRAIN:",i,"TEST",j)
TRAIN: [7 9 0 1 6 4 8 3] TEST [2 5]
TRAIN: [3 2 7 0 8 4 6 1] TEST [5 9]
TRAIN: [5 1 8 7 4 0 2 6] TEST [9 3]
TRAIN: [7 1 5 8 6 9 4 0] TEST [2 3]
TRAIN: [7 0 3 2 6 1 5 9] TEST [4 8]
user_43170580
Related