Send real time data to Django frontend

Viewed 1741

I've been making some questions about this topic but i still did not come to a conclusion and i'm still completely uncertain on what approach should i take here, so i decided to make a new question with all the details, in order to be as specific as possible.

The problem: i have an external Python script (we will call data-collector) that retrieves trades from various cryptocurrency markets in real time and prints them to my console; this script is up and running on its server and its printing trades, it works flawlessly. On the other side, i have a Django application. When the user opens a page on my Django application, they should receive the trades for a specific market. Example: user opens market ETHUSD > user needs to receive in real time the trades for ETHUSD from data-collector. So i need to find a way to make data collector send the trades to my Django application when the page is opened. I don't need to store that data on a database, i only need it to be seen by the user, nothing else.

POSSIBLE APPROACHES

1) Store on a Database and poll I thought of updating the data to a db and then, from Django, just query the database. This is not feasible, because, as i said, data-collector is retrieving trades from various markets every second, so it would result in a huge amount of queries both from server and client, also i don't want to store data, so i can't follow this path.

2) Pusher Pusher is maybe the best solution for this, but it would become, eventually, too expensive for me.

3) Redis PUB/SUB This was the most promising one: for every market, data-collector creates a Redis channel. On my Django application, when a user opens the page, a connection to the right channel is created and the data is retrieved. Example: user opens the ETHUSD page -> a connection to the Redis channel 'ETHUSD' is established and so the trades are received. The problem: creating a connection to a Redis channel from a Django Channels Consumer would result in a blocking operation, so it's not possible to do.

4) An abstract idea This one is an idea i had, but i don't know IF and HOW it would be feasible in terms of code. On data-collector i would create for every market a "channel" or a "room" using a websocket library. Then from the frontend on my Django application, i would just set up a connection to that 'channel' using Javascript. As i said, i don't know if this is something doable and how.

To make it as specific as possible, what i need is a solution to this problem. I need an approach to solve this problem that would work in production, if there is one. I'm willing to give more details, if needed. Since this problem is tormenting me, any kind of advice is highly appreciated!

My code:

data-collector.py

from binance.client import Client
import json
from binance.websockets import BinanceSocketManager

client = Client('', '')


# A list of the tickers, they are some hundreds
tickers = ['ETHBTC', 'BNBBTC', 'BATBTC', 'BTCUSDT' ....]

# This function is used to do something with the response,
# such as printing it 
def process_message(message):
    JSON1 = json.dumps(message)
    JSON2 = json.loads(JSON1)

    #define variables
    Rate = JSON2['p']
    Quantity = JSON2['q']
    Symbol = JSON2['s']
    Order = JSON2['m']

    print(Rate, Quantity, Order, Symbol)

bm = BinanceSocketManager(client)

# For every market, start the connection and stream the trades
for market in tickers:
    bm.start_trade_socket(market, process_message)

bm.start()

My basic Django Channels consumer

class EchoConsumer(SyncConsumer):

    def websocket_connect(self, event):
        self.send({
            'type': 'websocket.accept'
        })

        print('Connected')



    def websocket_receive(self, event):
        print('Received')

        self.send({
            'type': 'websocket.text',
            'text': event,
        })


    def websocket_disconnect(self, event):
        print('Disconnected')

My frontend websocket code

<script>
  // websocket scripts
  var loc = window.location
  var wsStart = 'ws://' + window.location.host + window.location.pathname
  console.log(wsStart)
  var endpoint = wsStart + loc.host + loc.pathname
  var socket = new WebSocket(endpoint)

  if (loc.protocol == 'https:'){
    wsStart = 'wss://'
  }

  socket.onmessage = function(e){
    console.log("message", e)
  }

  socket.onopen = function(e){
    console.log("message", e)
  }

  socket.onerror = function(e){
    console.log("message", e)
  }

  socket.onclose = function(e){
    console.log("message", e)
  }
</script>
1 Answers

Good to see we got similar ideas about designing this architectural issue. But i'm late. I wanted to share my approach.

I just started to build my personal project. Kind of real-time dashboard.

We know 1st and 2nd options are not worth of implementing. And i thought about 4th option too. Since clients receive similiar data, why we should group them to load faster. Anyway this is next step after implementing real-time data transfer.

Actually option 3 (Redis pub/sub) worked for me. As i write, my current project got window which is big as 5 sec to send data. And i'm going to update data every second.

(Unlike your project my frontend is Flutter. This shouldn't be problem.)

Related