I am attempting to plot a SHAP summary plot after training a Random Forest. I using VSCode to ssh into a remote worker and run my code there. When I run my code to generate and save the image, it does not throw any error, but only generates a white box/blank plot. This code is below (top). What's unusual is that when I test generating and saving figures with Matplotlib outside of SHAP, but on this ssh tunnel, it works totally fine (bottom).
I am using matplotlib version = 1.5.1 and SHAP version = 0.34.0. It is worth noting a few things about my attempts to debug this and my set up: (1) this exact code worked on a previous worker that I no longer have access to. To me, this suggests that it is a problem with the environment configuration; (2) I tried using plt.gcf() to get the SHAP plot into a purely matplotlib object and saving that, but this also did not work; (3) I have tried closing the plot (plt.close()) prior to the new plt.figure(), but this also did not work; (4) I have tried my current code both with and without the matplotlib=True flag; (5) I am trying to generate many SHAP figures (not shown), these also produced blank/white plots; (6) I am using a Mac, the remote worker is Linux (Ubuntu).
Any insights would be appreciated. Thanks.
CURRENT CODE -- RETURNS WHITE BOX/BLANK PLOT
import pandas as pd
import sklearn
import numpy as np
import os
import csv
import shap
import pickle
import warnings
import pydot
from sklearn import tree
from sklearn.tree import export_graphviz
from sklearn.utils import shuffle
from sklearn.model_selection import learning_curve, GridSearchCV, RandomizedSearchCV
from sklearn.model_selection import train_test_split
from sklearn.calibration import CalibratedClassifierCV
from sklearn.ensemble import RandomForestClassifier
from sklearn import metrics
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
#fit random forest here. very long, so I decided to omit it
plt.figure()
fig1 = shap.summary_plot(shap_values,tt, show=False, matplotlib=True, plot_type="bar", plot_size=(10, 10))
plt.savefig('/home/path/path/shap_summary.png', dpi=600)
plt.close()
MATPLOTLIB OUTSIDE OF SHAP -- SAVES A NORMAL PLOT
plt.figure()
x = np.linspace(0, 100, 1000)
plt.plot(x, np.log(x))
plt.savefig('home/path/thisworks.png')
plt.close()