Django app with Plotly/Dash callback and Celery/Redis

Question

I have a Django project where one of my apps provides interactive data visualization using Dash/Plotly and is deployed to Heroku. As my database grew bigger, the time it takes to collect the relevant data, normalize it, and plot it, is now often larger than 30s, therefore the requests are timing out and my plots are not showing up anymore, which is sad, because they were working great :/

What I'm trying to do now is to use Celery so that the data collection and normalization is dispatched to a worker and run asynchronously having Redis as a broker, but this is getting extremely confusing as I don't have any experience with these frameworks and I couldn't find an example of such an application. I'm posting below a simplified and analogous app without any Celery stuff, that works exactly how I want (the user provides a seed and the app plots a scatter plot with random numbers generated from that seed).

The project tree is:

djcel/
    djcel/
    worker/
        dash_apps/
            test_apps/
                plot.py
        templates/
            worker/
                scatter_plot.html
        urls.py
        views.py

And the main files are:

plot.py

from dash import dcc
from dash import html
from dash.dependencies import Input, Output
from django_plotly_dash import DjangoDash
import plotly.graph_objects as go

external_stylesheets = ['https://codepen.io/chriddyp/pen/bWLwgP.css']
app = DjangoDash('ScatterPlot', external_stylesheets=external_stylesheets)

def scatter_plot(seed):
    """
    Plots random data.
    """

    import numpy as np
    np.random.seed(seed=seed)
    x = np.arange(1,1000,1)
    y = np.random.rand(1000)

    fig = go.Figure(data=go.Scatter(x=x, y=y, mode='markers'))
        
    return fig



seeds = [11, 42, 97, 1001]

app.layout = lambda: html.Div([
    html.Div([
        html.Div([
            dcc.Dropdown(
                id='seed',
                options=[{'label': i, 'value': i} for i in seeds],
                placeholder='Select the seed...'
            )
        ],
        style={'width': '30%', 'display': 'inline-block'})
    ]), 
    
    dcc.Graph(id='Scatter-plot'),
])

@app.callback(
    Output('Scatter-plot', 'figure'),
    Input('seed', 'value')
)   
def update_graph(seed):
    if(seed):
        return scatter_plot(seed)
    else:
        return go.Figure()

views.py

from django.shortcuts import render
from plotly.offline import plot
from django.views.decorators.cache import cache_page

def scatter_plot_dispatch(requests):
    return render(requests, 'worker/scatter_plot.html')

urls.py

from django.urls import path
from . import views
from .dash_apps.test_apps import plot

urlpatterns = [
    path('scatterplot/', views.scatter_plot_dispatch, name="worker.scatter_plot"),
]

scatter_plot.html

{% extends "base.html" %}

{% block content %}
    {% load plotly_dash %}

    

    
        {% plotly_app name="ScatterPlot" ratio=0.85 %}
    

{% endblock %}

I would like to run scatter_plot asynchronously and have my view updated with the resulting figure object when the task is done. I know how to send tasks via Celery/Redis and how to check on their status, but I'm struggling to connect the pieces because of the DjangoDash callback. How could I make the callback return a message right away warning the user that it could take up to a minute to have the graph plotted and then have it automatically updated when the task is complete? Any ideas? I've been trying to figure this out for quite a while now. I'm happy to provide any further information. Thanks!!

Django app with Plotly/Dash callback and Celery/Redis

0 Answers