How to use Django ORM with Panel?

user_10608001 0
Viewed 161

Summary

I want to implement an interactive dashboard using panel in a Django application using data from a database.

First I've followed the example in the panel docs and it's working fine in my Django project. In that example, the data is calculated in a simple function and a plot is displayed. I can use the widgets and get the expected results.

For my application, instead of calculating the data without external dependencies, I want to aggregate data from a database depending on a date range which is chosen by the user in the UI. So I have to gather this data inside the param.Parametrized instance, as far as I understand.

Therefore I've tried to use a query on the database ORM instead of calculating values.

What I've tried

In order to keep it simple here and to stay with the example, I've created a model for the sine values in my Django app (called dashboard):

from django.db import models

class SineValue(models.Model):
    x = models.FloatField()
    y = models.FloatField()

and inserted some values:

DELETE FROM dashboard_sinevalue;
INSERT INTO dashboard_sinevalue (id,x,y) VALUES (1,1,1);
INSERT INTO dashboard_sinevalue (id,x,y) VALUES (2,2,4);
INSERT INTO dashboard_sinevalue (id,x,y) VALUES (3,3,9);

Then I've replaced the calculation of the sine values with code to request values from the database:

from .models import SineValue

[...]

class SineWave(param.Parameterized):
 
    [... same as in example ...]

    def sine(self):
        # OLD: calculate values        
        #x = np.linspace(0, 4 * np.pi, self.N)
        #y = self.amplitude * np.sin(self.frequency * x + self.phase) + self.offset

        # NEW: instead get from database
        all_sines = SineValue.objects.all()
        x,y = zip(*all_sines.values_list('x', 'y'))
        return x, y

Outside of panel, I can get the values without problems.

However, after changing the running example like this, I get this error:

django.core.exceptions.SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.

After a little research I've found out, that this is expected, since I try to execute synchronous code in a asynchronous environment. Django 3 has means for this: https://docs.djangoproject.com/en/3.1/topics/async/

So I've tried to use sync_to_async or even better database_sync_to_async, which seems to be made for this case. My problem is now, that the constructor

class SineWave(param.Parameterized):

    [... same as in example ...]

    def __init__(self, **params):
        super(SineWave, self).__init__(**params)
        x, y = self.sine()
        self.cds = ColumnDataSource(data=dict(x=x, y=y))
        self.plot = figure(plot_height=400, plot_width=400,
                           tools="crosshair, pan, reset, save, wheel_zoom",
                           x_range=self.x_range, y_range=self.y_range)
        self.plot.line('x', 'y', source=self.cds, line_width=3, line_alpha=0.6)

already calls the method self.sine(). So when I decorate the sine() method like this

from channels.db import database_sync_to_async

    [...]

    @database_sync_to_async
    def sine(self):
        all_sines = SineValue.objects.all()
        x,y = zip(*all_sines.values_list('x', 'y'))
        #x = np.linspace(0, 4 * np.pi, self.N)
        #y = self.amplitude * np.sin(self.frequency * x + self.phase) + self.offset
        return x, y

I need to call it as coroutine everywhere, so I thought also in the constructor using await:

class SineWave(param.Parameterized):

    [...]

    def __init__(self, **params):
        super(SineWave, self).__init__(**params)
        x, y = await self.sine()
        [...]

which is not possible:

SyntaxError: 'await' outside async function

I thought this would be a common task when using panel, but I haven't found a solution yet. This discussion seems related and I did some experiments using from concurrent.futures import ThreadPoolExecutor as described in the discussion. At the end I've got the message I've started with:

You cannot call this from an async context - use a thread or sync_to_async

Questions

So my questions here are:

  • Is it possible to use results of Django ORM database queries in Panel widgets?
  • How to change the example for this? Is the decorator @database_sync_to_async the way to go or sth else?
  • Is it possible to trigger new database requests if a widget changes, e.g. for putting a filter on the queryset?
  • If this is not possible, do you have other ideas how to implement an interactive dashboard with Django, such that I can display results from database requests?

Concerning async/await, I'm a beginner. Thanks for any hints.

asynchronous panel django python
1 Answers

Ok I managed to solve this with a relatively simple solution

In my code I'm not returning anything from the ORM function I am just manipulating self variables and I am happy for it to run async (not waiting for it to finish) so I run it like...

import asyncio

def __init__(self, **params):
    super(SineWave, self).__init__(**params)
    coroutine = self.sine()
    asyncio.create_task(coroutine)

@database_sync_to_async
def sine(self):
    all_sines = SineValue.objects.all()
    x,y = zip(*all_sines.values_list('x', 'y'))
    self.x = np.linspace(0, 4 * np.pi, self.N)
    self.y = self.amplitude * np.sin(self.frequency * x + self.phase) + self.offset
    # return x, y

If you need the results returned syncronously you can try

def __init__(self, **params):
    super(SineWave, self).__init__(**params)
    coroutine = self.sine()
    task = asyncio.create_task(coroutine)
    x,y = task.result()

More context is here https://docs.python.org/3/library/asyncio-task.html#task-object basically what this is doing is adding async tasks to the current event loop that the bokeh app is running in.

user_81171330
Related