Preventing infinite callback loop deploying a Dash app with Gunicorn with `dcc.Interval` and `dcc.Location` (i.e., multi-page app) components present

user_9784373 0
Viewed 1109

I am using Dash 2.0 and I have interval.py inside apps folder:

import numpy as np
import dash
import pandas as pd
import plotly.express as px
from dash.dependencies import Output, Input
from dash import dcc
from dash import html

external_stylesheets = ['https://codepen.io/chriddyp/pen/bWLwgP.css']
app = dash.Dash(__name__, external_stylesheets=external_stylesheets)  # call flask server


app.layout = html.Div(className='row', children=[
    html.Div(children=[dcc.Graph(id='a-graph', style={'display': 'flex'}),
                       dcc.Graph(id='b-graph', style={'display': 'flex'}),
                       dcc.Interval(id='interval-component', interval=1000)])])


@app.callback(
    [
        Output('a-graph', 'figure'),
        Output('b-graph', 'figure')
    ],
    [
        Input('interval-component', 'n_intervals')
    ]
)
def update_graphs(n):
    print(f'callback happened: {n}')
    df = pd.DataFrame({"time":pd.Series(pd.date_range("1-nov-2021","2-nov-2021", freq="S")).sample(30), "bacteria_count":np.random.randint(0,500, 30), "bacteria_type":np.random.choice(list("AB"),30)})

    df["epoch_time_ms"] = df["time"].astype(int) / 1000
    df = df.sort_values("time")
    a_fig = px.line(df, x="time", y="bacteria_count", line_shape="hv", markers=True, color='bacteria_type')
    a_fig.update_traces(mode="markers+lines", hovertemplate=None)
    a_fig.update_layout(hovermode='x unified')

    b_fig = px.line(df, x="epoch_time_ms", y="bacteria_count", line_shape="hv", markers=True, color='bacteria_type')
    b_fig.update_traces(mode="markers+lines", hovertemplate=None)
    b_fig.update_layout(hovermode='x unified')
    return a_fig, b_fig


if __name__ == '__main__':
    app.run_server(debug=True)

update_graphs() gets called periodically as expected if I run python3 interval.py.

However, if I add an index page (index.py) like this:

from dash import dcc
from dash import html
from dash.dependencies import Input, Output

from app import app
from apps import interval

app.layout = html.Div([
    # represents the URL bar, doesn't render anything
    dcc.Location(id='url', refresh=False),

    dcc.Link('Top | ', href='/'),
    dcc.Link('Interval | ', href='/apps/interval'),

    # content will be rendered in this element
    html.Div(id='page-content')
])

server = app.server


@app.callback(Output('page-content', 'children'),
              Input('url', 'pathname'))
def display_page(pathname):
    if pathname == '/apps/interval':
        return interval.app.layout
    else:
        return ''


if __name__ == '__main__':
    app.run_server(debug=True)

and app.py:

import dash

app = dash.Dash(__name__, suppress_callback_exceptions=True)
server = app.server

Then I started it with gunicorn index:server -b :8086. It started without any error even when I loaded the interval page. However, the callback never gets called.

Then, I tried to replace app = dash.Dash(__name__, external_stylesheets=...) in interval.py with from app import app. Once I click on the /apps/interval link on index page, it gets into an infinite loop -- keep printing Top | Interval on my browser.

Any idea?

Thanks!

gunicorn plotly-dash python
1 Answers

Deploying a Dash app with gunicorn with Dash dcc.Interval† as well as dcc.Location⁂ (i.e., multi-page app) features present*

* Refs to Dash Docs for mentioned components:

† Interval dcc (dash core components) components allow for auto source data updating (and thus auto-updates for certain components on a page) at specified intervals.

⁂ Location dcc components allow for access to the current href/pathname/URL and thus enabling of multi-page Dash apps.

[Minimal] Code Setup

So I got this to work, based off what you provide - though you'll surely have some more work to do carrying off where I leave you (and hopefully this makes sense; there are multiple ways of doing this). BTW, I set max_intervals=5 for the dcc.Interval component because otherwise it just updated the graphs endlessly (perhaps that's the behavior you wanted, but, setting this limit just helps to simplify this demo).

Here's the file structure I arrived at, followed by the exact contents of each file.

.
|-- app
|   |-- __init__.py
|   |-- gunicorn
|   |   `-- logs
|   |       `-- 20211101
|   |           `-- 20211101.access.log
|   `-- interval.py
|-- index.py
`-- launch_gunicorn.sh

4 directories, 5 files

Sticking with the approach you were most closely going after, which was to essentially have the layout, callbacks, and app all declared in a single file — "interval.py". Then, there is simply just one additional Python file "index.py" (again keeping in conformity with your post) which controls the multi-page layout handling of the app, and is also the file from which gunicorn will obtain its required reference to the Dash app object (specifically the server sub-attribute of the app; i.e., in this case, that'd be app.interval.app.server). Finally there is just one other third extra file I created which is simply a bash launch_gunicorn.sh shell script which facilitates execution of the non-development/debug (i.e. production [although may want to add nginx too but that's another question]) deployment of the app.

1) app/interval.py

import dash
import numpy as np
import pandas as pd
import plotly.express as px

from dash import dcc
from dash import html
from dash.dependencies import Input
from dash.dependencies import Output

external_stylesheets = ["https://codepen.io/chriddyp/pen/bWLwgP.css"]
app = dash.Dash(__name__, external_stylesheets=external_stylesheets)

interval_layout = html.Div(
    className="row",
    children=[
        html.Div(
            children=[
                html.Div(id="interval-update"),
                dcc.Graph(id="a-graph", style={"display": "flex"}),
                dcc.Graph(id="b-graph", style={"display": "flex"}),
                dcc.Interval(
                    id="interval-component", interval=1000, max_intervals=5
                ),
            ]
        )
    ],
)


@app.callback(
    [
        Output("a-graph", "figure"),
        Output("b-graph", "figure"),
        Output("interval-update", "children"),
    ],
    [Input("interval-component", "n_intervals")],
)
def update_graphs(n):
    app.logger.info(f"callback happened: {n}")
    df = pd.DataFrame(
        {
            "time": pd.Series(
                pd.date_range("1-nov-2021", "2-nov-2021", freq="S")
            ).sample(30),
            "bacteria_count": np.random.randint(0, 500, 30),
            "bacteria_type": np.random.choice(list("AB"), 30),
        }
    )

    df["epoch_time_ms"] = df["time"].astype(int) / 1000
    df = df.sort_values("time")
    a_fig = px.line(
        df,
        x="time",
        y="bacteria_count",
        line_shape="hv",
        markers=True,
        color="bacteria_type",
    )
    a_fig.update_traces(mode="markers+lines", hovertemplate=None)
    a_fig.update_layout(hovermode="x unified")

    b_fig = px.line(
        df,
        x="epoch_time_ms",
        y="bacteria_count",
        line_shape="hv",
        markers=True,
        color="bacteria_type",
    )
    b_fig.update_traces(mode="markers+lines", hovertemplate=None)
    b_fig.update_layout(hovermode="x unified")
    interval_update = [html.H1(f"Interval Update - {n}")]
    return a_fig, b_fig, interval_update


if __name__ == "__main__":
    app.run_server(debug=True)

2) index.py

""" ## Primary Application Deployment Script
------------------------------------

>        
_____

ᵂʳⁱᵗᵗᵉⁿ ᵇʸ
  [your name]


Purpose / Overview:
------------------
This script serves as the de facto WSGI Production-level deployment module (in this case, fed to Gunicorn).

For Development-level deployment (DEBUG),
simply run this script, like so:

$ python index.py

    :   :   : ::::::: :   :   :

"""
import logging

from dash import dcc
from dash import html
from dash import no_update

from dash.dependencies import Input
from dash.dependencies import Output
from dash.dependencies import State

from dash.exceptions import PreventUpdate

from app import interval
from app.interval import app

app.layout = html.Div(
    [
        # represents the URL bar, doesn't render anything
        dcc.Location(id="url", refresh=False),
        dcc.Link("Top | ", href="/"),
        dcc.Link("Interval | ", href="/apps/interval"),
        # content will be rendered in this element
        html.Div(id="page-content"),
    ]
)


@app.callback(Output("page-content", "children"), Input("url", "pathname"))
def display_page(pathname):
    if pathname == "/apps/interval":
        return interval.interval_layout
    elif pathname == "/":
        return no_update
    else:
        return html.Div(
            [html.H1("404 Page Not Found")],
            style={"margin": "10%", "textAlign": "center"},
        )


if __name__ == "__main__":
    app.run_server(debug=True)


############################################
# #       DEPLOY* MODE: PRODUCTION       # #
# # [*WSGI import (e.g. Gunicorn+Nginx)] # #
############################################
if __name__ != "__main__":
    logger = logging.getLogger(__name__)
    logging.getLogger("matplotlib.font_manager").disabled = True
    logging.basicConfig(
        format="%(asctime)s %(name)-12s %(module)s.%(funcName)s %(processName)s %(levelname)-8s %(relativeCreated)d %(message)s",
        level=logging.INFO,
    )
    gunicorn_logger = logging.getLogger("gunicorn.error")
    app.logger.handlers = logger.handlers
    app.logger.setLevel(logger.level)
    app.logger.info(
        "Initializing dash-webapp-template (App) `app.server` for handoff..."
    )
    server = app.server

3) launch_gunicorn.sh

#!/bin/bash

# Gunicorn Launch script for Flask-based Dash app,
# ran concurrently, & in parallel.

TODAY=$(date "+%Y%m%d")
TIMESTAMP=$(date | tr -d "[[:punct:]]" | tr -d ' ')

PORT=${1:-9001}
NUM_WORKERS=${2:-4}
THREADS=${3:-8}

GUNICORN_PROD_LOGS="./app/gunicorn/logs/${TODAY}"
mkdir -p $GUNICORN_PROD_LOGS

gunicorn \
  -b 0.0.0.0:$PORT \
  -w $NUM_WORKERS \
  --worker-class gthread \
  --threads $THREADS \
  --name "dash-webapp-demo${TODAY}" \
  --max-requests 100 \
  --max-requests-jitter 10 \
  --access-logfile "${GUNICORN_PROD_LOGS}/${TODAY}.access.log" \
  index:server

Deploying the app via Gunicorn

Upon launching launch_gunicorn.sh, you should see the following output being printed in your terminal:

[...]$ ./launch_gunicorn.sh 
[2021-11-01 22:51:28 -0700] [23576] [INFO] Starting gunicorn 20.1.0
[2021-11-01 22:51:28 -0700] [23576] [INFO] Listening at: http://0.0.0.0:9001 (23576)
[2021-11-01 22:51:28 -0700] [23576] [INFO] Using worker: gthread
[2021-11-01 22:51:28 -0700] [23580] [INFO] Booting worker with pid: 23580
[2021-11-01 22:51:28 -0700] [23581] [INFO] Booting worker with pid: 23581
[2021-11-01 22:51:28 -0700] [23582] [INFO] Booting worker with pid: 23582
[2021-11-01 22:51:28 -0700] [23583] [INFO] Booting worker with pid: 23583
2021-11-01 22:51:30,464 app.interval index.<module> MainProcess INFO     2686 Initializing dash-webapp-template (App) `app.server` for handoff...
2021-11-01 22:51:30,489 app.interval index.<module> MainProcess INFO     2712 Initializing dash-webapp-template (App) `app.server` for handoff...
2021-11-01 22:51:30,560 app.interval index.<module> MainProcess INFO     2783 Initializing dash-webapp-template (App) `app.server` for handoff...
2021-11-01 22:51:30,621 app.interval index.<module> MainProcess INFO     2844 Initializing dash-webapp-template (App) `app.server` for handoff...

This is because I have set as default the number of workers to 4. You can change this in the launch_gunicorn.sh file, or, pass as arguments. As you can see, the script accepts three optional arguments: the port, number of workers, and number of threads, respectively.

We are then presented with the following very basic main app page:

main app page

upon which clicking the "Interval" link gives:

interval 5

which was the fifth and final automatic "interval" page update via the Dash dcc.Interval component in the main interval.py app layout + callbacks (you did an all-in-one approach - which works fine!) file.

Monitoring the app during active deployment

We can see back in the terminal the following updates logged:

2021-11-01 23:11:26,140 app.interval interval.update_graphs MainProcess INFO     175805 callback happened: None
2021-11-01 23:11:26,190 numexpr.utils utils._init_num_threads MainProcess INFO     175854 NumExpr defaulting to 4 threads.
2021-11-01 23:11:27,270 app.interval interval.update_graphs MainProcess INFO     176935 callback happened: 1
2021-11-01 23:11:28,131 app.interval interval.update_graphs MainProcess INFO     177796 callback happened: 2
2021-11-01 23:11:29,129 app.interval interval.update_graphs MainProcess INFO     178793 callback happened: 3
2021-11-01 23:11:30,136 app.interval interval.update_graphs MainProcess INFO     179801 callback happened: 4
2021-11-01 23:11:31,133 app.interval interval.update_graphs MainProcess INFO     180797 callback happened: 5

And in the app/gunicorn/logs/[today's date]/[today's date].access.log file automatically created, you'll find information such as:

127.0.0.1 - - [01/Nov/2021:23:08:43 -0700] "GET / HTTP/1.1" 200 1781 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"
127.0.0.1 - - [01/Nov/2021:23:08:43 -0700] "GET /_dash-layout HTTP/1.1" 200 462 "http://0.0.0.0:9001/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"
127.0.0.1 - - [01/Nov/2021:23:08:43 -0700] "GET /_dash-dependencies HTTP/1.1" 200 357 "http://0.0.0.0:9001/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"
127.0.0.1 - - [01/Nov/2021:23:08:44 -0700] "POST /_dash-update-component HTTP/1.1" 204 0 "http://0.0.0.0:9001/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"
127.0.0.1 - - [01/Nov/2021:23:08:53 -0700] "GET /_dash-component-suites/dash/dcc/dash_core_components-shared.js.map HTTP/1.1" 200 26038 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"
127.0.0.1 - - [01/Nov/2021:23:08:53 -0700] "GET /_dash-component-suites/dash/html/dash_html_components.min.js.map HTTP/1.1" 200 687871 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"
127.0.0.1 - - [01/Nov/2021:23:08:53 -0700] "GET /_dash-component-suites/dash/dash_table/bundle.js.map HTTP/1.1" 200 154018 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"
127.0.0.1 - - [01/Nov/2021:23:08:53 -0700] "GET /_dash-component-suites/dash/dcc/dash_core_components.js.map HTTP/1.1" 200 1996412 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"
127.0.0.1 - - [01/Nov/2021:23:11:26 -0700] "POST /_dash-update-component HTTP/1.1" 200 651 "http://0.0.0.0:9001/apps/interval" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"
127.0.0.1 - - [01/Nov/2021:23:11:26 -0700] "GET /_dash-component-suites/dash/dcc/async-graph.js HTTP/1.1" 304 0 "http://0.0.0.0:9001/apps/interval" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"
127.0.0.1 - - [01/Nov/2021:23:11:26 -0700] "GET /_dash-component-suites/dash/dcc/async-graph.js.map HTTP/1.1" 200 75008 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"
127.0.0.1 - - [01/Nov/2021:23:11:26 -0700] "GET /_dash-component-suites/dash/dcc/async-plotlyjs.js HTTP/1.1" 200 3594037 "http://0.0.0.0:9001/apps/interval" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"
127.0.0.1 - - [01/Nov/2021:23:11:27 -0700] "POST /_dash-update-component HTTP/1.1" 200 16924 "http://0.0.0.0:9001/apps/interval" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"
127.0.0.1 - - [01/Nov/2021:23:11:27 -0700] "POST /_dash-update-component HTTP/1.1" 200 16915 "http://0.0.0.0:9001/apps/interval" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"
127.0.0.1 - - [01/Nov/2021:23:11:28 -0700] "POST /_dash-update-component HTTP/1.1" 200 16909 "http://0.0.0.0:9001/apps/interval" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"
127.0.0.1 - - [01/Nov/2021:23:11:29 -0700] "POST /_dash-update-component HTTP/1.1" 200 16911 "http://0.0.0.0:9001/apps/interval" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"
127.0.0.1 - - [01/Nov/2021:23:11:30 -0700] "POST /_dash-update-component HTTP/1.1" 200 16903 "http://0.0.0.0:9001/apps/interval" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"
127.0.0.1 - - [01/Nov/2021:23:11:31 -0700] "POST /_dash-update-component HTTP/1.1" 200 16915 "http://0.0.0.0:9001/apps/interval" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"

showing the requests received by the server.

Finally, running e.g. htop and filtering down (in this case I just filtered on "dash", because in the launch_gunicorn.sh file in the gunicorn command options there is config for giving a specific name to your gunicorn worker processes; see below) for our app we can see the actual gunicorn workers:

gunicorn workers

Lmk if any questions/issues

user_67146270
Related