The reason why your example doesn't work is that in Dash, callbacks must be registered before the server starts. Hence, you cannot register new callbacks from within a callback.
Data pre-processing pipeline
I think the cleanest solution would be move the data processing to a pre-processing pipeline. It could be something as simple as a notebook running on the Dataiku node. The code would be along the lines of
from explainerdashboard import ClassifierExplainer
from explainerdashboard.datasets import titanic_survive
from sklearn.linear_model import LogisticRegression
X_train, y_train, X_test, y_test = titanic_survive()
model = LogisticRegression().fit(X_train, y_train)
explainer = ClassifierExplainer(model, X_test, y_test)
explainer.dump("/data/dataiku/titanic.joblib") # save to some writeable location
The corresponding webapp code would be something like,
import dash_bootstrap_components as dbc
from dash import Dash
from explainerdashboard import ClassifierExplainer, ExplainerDashboard
explainer = ClassifierExplainer.from_file("/data/dataiku/titanic.joblib") # load pre-processed data
db = ExplainerDashboard(explainer, shap_interaction=False)
app.config.external_stylesheets = [dbc.themes.BOOTSTRAP]
app.layout = db.explainer_layout.layout()
db.explainer_layout.register_callbacks(app)
The deployment process would then be to (1) run the notebook and (2) (re)start the webapp backend. Note that this process must be repeated for the app to pickup new data.
Callback registration using mock data
Another approach could be to use a mock dataset that is small, but has the same structure as your normal (large) dataset, for constructing the ExplainerDashboard during app initialisation. This approach enables fast initial loading, and callback registration before app start. You could then use a callback to load the complete dataset afterwards, i.e. similar to your original idea. Here is some example code,
import dash_bootstrap_components as dbc
from dash import html, Dash, Output, Input, dcc
from dash.exceptions import PreventUpdate
from explainerdashboard import ClassifierExplainer, ExplainerDashboard
from explainerdashboard.datasets import titanic_survive
from sklearn.linear_model import LogisticRegression
def get_explainer(X_train, y_train, X_test, y_test, limit=-1):
model = LogisticRegression().fit(X_train[:limit], y_train[:limit])
return ClassifierExplainer(model, X_test[:limit], y_test[:limit])
def inject_inplace(src, dst):
for attr in dir(dst):
try:
setattr(dst, attr, getattr(src, attr))
except AttributeError:
pass
except NotImplementedError:
pass
X_train, y_train, X_test, y_test = titanic_survive()
# Create explainer with minimal data to ensure fast initial load.
explainer = get_explainer(X_train, y_train, X_test, y_test, limit=5)
dashboard = ExplainerDashboard(explainer, shap_interaction=False)
# Setup app with (hidden) dummy classifier layout.
dummy_layout = html.Div(dashboard.explainer_layout.layout(), style=dict(display="none"))
app = Dash() # not needed in Dataiku
app.config.external_stylesheets = [dbc.themes.BOOTSTRAP]
app.layout = html.Div([
html.Button('Submit', id='submit', n_clicks=0),
dcc.Loading(html.Div(id='container', children=dummy_layout), fullscreen=True)
])
# Register the callback before the app starts.
dashboard.explainer_layout.register_callbacks(app)
@app.callback(Output('container', 'children'), Input('submit', 'n_clicks'))
def load_complete_dataset(n_clicks):
if n_clicks != 1:
raise PreventUpdate
# Replace in-memory references to the full dataset to sure callbacks target the full dataset.
full_explainer = get_explainer(X_train, y_train, X_test, y_test)
inject_inplace(full_explainer, explainer)
return ExplainerDashboard(explainer, shap_interaction=False).explainer_layout.layout()
if __name__ == "__main__":
app.run_server(port=9024, debug=False)
