Index 38 is out of bounds for axis 1 with size 38 - Sklearn

user_11433497 0
Viewed 2041

I am facing this error with the Naive Bayes's CategoricalNB algorithm

It gives the above error after the 2nd attempt I run the cells. That means it works without any errors during the 1st time and when I try to change something (as small as a comment) and run the notebook again, it gives the error:

IndexError: index 38 is out of bounds for axis 1 with size 38

I don't know what is going wrong and how to fix. When I restart the kernel and try again it works and every attempt after the 1st attempt it fails and gives the above error.

%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd

dataframe = pd.read_csv("hr_dataset.csv")
# dataframe = pd.read_csv("WA_Fn-UseC_-HR-Employee-Attrition.csv")

dataframe.head(2)

from sklearn.naive_bayes import CategoricalNB
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
# inputs = scaled_df
X_train, X_test, y_train, y_test = train_test_split(inputs, target, test_size=0.2)

categoricalNB_ = CategoricalNB()


categoricalNB_.fit(X_train, y_train)
X_train.shape, X_test.shape, y_train.shape, y_test.shape


pred = categoricalNB_.predict(X_test) # --------------> gives the error for every attempt after the 1st attempt. weird

categoricalNB_.score(X_test, y_test)
# accuracy_score(y_test,pred)
scikit-learn pandas python
1 Answers

I believe your issue is related to having a different set of values for a feature in the train, and feature set.

I went through your database, and found that you have only one record, where the total working years is 38. If that record is only accessible in the test set, then your fit from the training set will not contain a probability for the value 38 thus throwing an out of bounds error.

You could either solve this with class_prior parameter (for more details read docs), or you could make sure, that you have for every feature every category at least a certain amount of record.

user_88590920
Related