Getting UnicodeDecodeError when using shap on xgboost

Viewed 2687

I'm trying to use shap on xgboost model, but getting error:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 341: invalid start byte

example:

model = XGBClassifier()
model.fit(X_train, y_train)
explainer = shap.TreeExplainer(model)

package versions:

    python == 3.6.9
    xgboost==1.1.0
    shap==0.35.0

What is the problem and how can we fix it ?

4 Answers

I tried the following solution and it worked.

package versions:
    python == 3.7.7
    xgboost==1.1.1
    shap==0.35.0

The code has worked very well for me.

import shap
from xgboost.sklearn import XGBClassifier

xgb = XGBClassifier(random_state=42)
mymodel = xgb.fit(X_train, y_train)

The part that really solves them is this, must not miss

mybooster = mymodel.get_booster()    
model_bytearray = mybooster.save_raw()[4:]
def myfun(self=None):
    return model_bytearray
mybooster.save_raw = myfun

# Shap explainer initilization
shap_ex = shap.TreeExplainer(mybooster)

There seems to be a bug in the system. See: https://github.com/slundberg/shap/issues/1215. The issue seems to be resolved, but perhaps the fix is not yet released. At any rate, I had the same issue and resolved it temporarily by installing xgboost v1.0.0.

I had the same problem with xgboost-1.2.0 and shap 0.35.0.

Here is the complete example that I was able to run without problem:

import numpy as np
import xgboost as xgb
import shap

# data
np.random.seed(100)
X_train = np.random.random((100, 10))
y_train = np.random.randint(2, size=100)

# model
model = xgb.XGBClassifier(random_state=42)
fitted_model = model.fit(X_train, y_train)

# monkey patch
booster = fitted_model.get_booster() 
model_bytearray = booster.save_raw()[4:]
booster.save_raw = lambda : model_bytearray

# shap expaliner
explainer = shap.TreeExplainer(booster)
shap_values = explainer.shap_values(X_train)
shap.summary_plot(shap_values, X_train)

output

enter image description here

!pip install shap==0.36.0
!pip install xgboost==1.3.3

This worked for me well

Related