How can I combine a Django REST backend and an event-driven backend?

NeonCop 0
Viewed 33

I am developing an app that's going to have two capabilities. First of all, it has to be able to ingest a huge load of events (think millions per minute). It's also going to have a typical REST frontend/backend where I'll have the classic authentication flows, some dashboard based on an analysis of the ingested events, etc. For the REST portion of the app, I'm using Django + Postgres and React on Docker. For the event-driven backend, since Django is too slow to handle clients sending me millions of requests, I was thinking of using Kafka + a streaming database like Materialize.

However, I'm still unclear on how I would route requests to these different endpoints. Do I need to write an endpoint in something fast like Golang/Rust to send client payloads to Kafka? Or do clients communicate with Kafka directly? Is something like Nginx the right way to route these requests? I would still need to have some sort of reference to the the Postgres DB in order to verify the API Key used in the client request is valid.

event-driven apache-kafka nginx django-rest-framework docker
1 Answers

be able to ingest a huge load of events (think millions per minute).

Through HTTP, or Kafka? If Kafka, then you can use Kafka Connect to directly write into the database without needing any frontend web server.

Django is too slow to handle clients sending me millions of requests

Based on your own benchmarks? Using how many instances?

Do I need to write an endpoint in something fast like Golang/Rust to send client payloads to Kafka?

Python can send producer requests to Kafka. There's nothing preventing you from using Kafka libraries in Django, although none of your question seems very specific to Kafka vs other message queues, especially when you've only referenced needing one or two databases.

If you're going to pick a different "service worker" language, you may as well write your HTTP server using those as well...

Or do clients communicate with Kafka directly?

Web clients? No, at least not without an HTTP proxy.

Is something like Nginx the right way to route these requests?

Unclear what you mean by "requests" here. Kafka requests - no, database requests - probably not, load balancing of HTTP requests - yes.

have some sort of reference to the Postgres DB in order to verify the API Key used in the client request is valid

Okay fine, use a Postgres client in your backend code.

OneCricketeer0
Related