I am trying to apply LIME to a Deep Learning approach I found in an interesting book (Deep Learning with Structured Data). To go about that I downloaded the repo from github (see here) and started familiarizing with the code in the streetcar_model_training.ipynb (after reading the book) and the documentation of the LIME package. After all that and some testing I am not able to generate an explanation via explain_instance() method that LIME provides. I am always getting a KeyError and I don't really get why that is the case. Can anyone help me figure out what exactly I am doing wrong?
My added code to the notebook mentioned above is:
#convert datetime to int for LIME
dtrain['Report Date'] = dtrain['Report Date'].apply(lambda x: x.value)
test['Report Date'] = test['Report Date'].apply(lambda x: x.value)
because else it was throwing an error that LIME could not handle the timestamp format that this variable was saved as. After that I ran:
import lime
from lime import lime_tabular
explainer = lime.lime_tabular.LimeTabularExplainer(dtrain.values, mode='classification', feature_names=dtrain.columns.values.tolist(), class_names=dtrain['target'].unique().tolist(), verbose=True)
#explain specific instance
exp = explainer.explain_instance(test.values[1], model.predict, num_features=10)
The resulting error message is:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
Input In [68], in <cell line: 2>()
1 #explain specific instance
----> 2 exp = explainer.explain_instance(test.values[1], model.predict, num_features=10)
File D:\Programme\Anaconda\envs\Masterarbeit\lib\site-packages\lime\lime_tabular.py:340, in LimeTabularExplainer.explain_instance(self, data_row, predict_fn, labels, top_labels, num_features, num_samples, distance_metric, model_regressor)
337 if sp.sparse.issparse(data_row) and not sp.sparse.isspmatrix_csr(data_row):
338 # Preventative code: if sparse, convert to csr format if not in csr format already
339 data_row = data_row.tocsr()
--> 340 data, inverse = self.__data_inverse(data_row, num_samples)
341 if sp.sparse.issparse(data):
342 # Note in sparse case we don't subtract mean since data would become dense
343 scaled_data = data.multiply(self.scaler.scale_)
File D:\Programme\Anaconda\envs\Masterarbeit\lib\site-packages\lime\lime_tabular.py:540, in LimeTabularExplainer.__data_inverse(self, data_row, num_samples)
538 inverse = data.copy()
539 for column in categorical_features:
--> 540 values = self.feature_values[column]
541 freqs = self.feature_frequencies[column]
542 inverse_column = self.random_state.choice(values, size=num_samples,
543 replace=True, p=freqs)
KeyError: 11