"KeyError: 0" with xgboost, scikit-learn and pandas

user_2905013 0
Viewed 3410

I created this demo to demonstrate an error thrown from the library internals. This code splits dataset into train/eval/test and uses train/eval for hyper-parameter search, early stopping, while the test set is hold out for later evaluation. I narrowed down the error to be related to cross validation of GridSearchCV but I couldn't figure out the exact root cause and fix.

from sklearn import svm, datasets
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import train_test_split
import numpy as np
import pandas as pd
import xgboost as xgb

iris = datasets.load_iris()
df = pd.DataFrame(data=np.c_[iris['data'], iris['target']], columns=iris['feature_names'] + ['target'])
X, y = df[['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']], df['target']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_train_split, X_eval_split, y_train_split, y_eval_split = train_test_split(X_train, y_train, test_size=0.25, random_state=42)

parameters = {
    'max_depth': (2, 3, 4),
}

fit_params = {
    'early_stopping_rounds': 2,
    'eval_set': (X_eval_split, y_eval_split),
}

model = xgb.XGBClassifier()
gs = GridSearchCV(model, parameters, cv=3)
gs.fit(X_train_split, y_train_split, **fit_params)

However I'm receiving this obscure message:

Traceback (most recent call last):
  File "/Users/foo/bar/.env/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 3078, in get_loc
    return self._engine.get_loc(key)
  File "pandas/_libs/index.pyx", line 140, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 162, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1492, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1500, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "t.py", line 36, in <module>
    gs.fit(X_train_split, y_train_split, **fit_params)
  File "/Users/foo/bar/.env/lib/python3.6/site-packages/sklearn/model_selection/_search.py", line 640, in fit
    cv.split(X, y, groups)))
  File "/Users/foo/bar/.env/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py", line 779, in __call__
    while self.dispatch_one_batch(iterator):
  File "/Users/foo/bar/.env/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py", line 625, in dispatch_one_batch
    self._dispatch(tasks)
  File "/Users/foo/bar/.env/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py", line 588, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "/Users/foo/bar/.env/lib/python3.6/site-packages/sklearn/externals/joblib/_parallel_backends.py", line 111, in apply_async
    result = ImmediateResult(func)
  File "/Users/foo/bar/.env/lib/python3.6/site-packages/sklearn/externals/joblib/_parallel_backends.py", line 332, in __init__
    self.results = batch()
  File "/Users/foo/bar/.env/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py", line 131, in __call__
    return [func(*args, **kwargs) for func, args, kwargs in self.items]
  File "/Users/foo/bar/.env/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py", line 131, in <listcomp>
    return [func(*args, **kwargs) for func, args, kwargs in self.items]
  File "/Users/foo/bar/.env/lib/python3.6/site-packages/sklearn/model_selection/_validation.py", line 458, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/Users/foo/bar/.env/lib/python3.6/site-packages/xgboost/sklearn.py", line 526, in fit
    for i in range(len(eval_set))
  File "/Users/foo/bar/.env/lib/python3.6/site-packages/xgboost/sklearn.py", line 526, in <genexpr>
    for i in range(len(eval_set))
  File "/Users/foo/bar/.env/lib/python3.6/site-packages/pandas/core/frame.py", line 2688, in __getitem__
    return self._getitem_column(key)
  File "/Users/foo/bar/.env/lib/python3.6/site-packages/pandas/core/frame.py", line 2695, in _getitem_column
    return self._get_item_cache(key)
  File "/Users/foo/bar/.env/lib/python3.6/site-packages/pandas/core/generic.py", line 2489, in _get_item_cache
    values = self._data.get(item)
  File "/Users/foo/bar/.env/lib/python3.6/site-packages/pandas/core/internals.py", line 4115, in get
    loc = self.items.get_loc(item)
  File "/Users/foo/bar/.env/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 3080, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx", line 140, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 162, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1492, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1500, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 0

Could someone help with some pointers about why I received this error?

xgboost scikit-learn pandas python
2 Answers

According to the documentation:

eval_set (list, optional) – A list of (X, y) tuple pairs to use as a validation set for early-stopping

eval_set should be a list of tuples. But you have eval_set as only a tuple:

fit_params = {
    'early_stopping_rounds': 2,
    'eval_set': (X_eval_split, y_eval_split),
}

Change that to this:

fit_params = {
    'early_stopping_rounds': 2,
    'eval_set': [(X_eval_split, y_eval_split)],
}

and your code will run.

user_33749960

As I read through the error trace, I see that the fit method is having some problem.

KeyError : 0 means that the interpreter is looking for an element/item in the data frame that is in 0th indexed position. I tried running your X_train_split,y_train_split and X_eval_split. The indices are different and possibly it is breaking the execution.

However, the purpose of cross validation might get beaten, if we don't shuffle dataset for training and evaluation.

Try resetting the index of things in fit method including evaluation(as its used in parameters) . If issue persists, read through concept of early stopping parameters, train test split and Gridsearch cv =3 areas to check for any inconsistency .

Hope it gives you some idea on the error.

user_99686770
Related