Query a Kafka Topic with nodejs

user_2815264 0
Viewed 299

I'm a bit puzzled. Is there really no NodeJS API to query Kafka topics (e.g. as with Kafka Streams and the Java API)? Am I missing something?

Just to get this straight. Only be notified of the latest event/record of a topic is not enough for me. I want to query and process the topics' records - and then maybe store an aggregate to another topic.

thanks for your thoughts if this is possible with nodejs and a library only.

apache-kafka nestjs express node.js
1 Answers

Here what worked for me and most people use.

Limited solution

If you are stubborn and want to insist on a node.js library to wrap things up for you: https://nodefluent.github.io/kafka-streams/docs/

As of today they offer:

  • easy access streams
  • merge streams
  • split streams

Full blown solution

The easiest way (as from today - there are rumors Confluent is creating more libraries and also one for nodejs) one can query kafka is by the REST API. It is part of the ksqlDB and ksqlDB is part of the confluent platform which also ships with Zookeeper and a Kafka instance, which you probably already have. If you wonder how to install:

  1. It spins up in 1 minutes with the docker yml file.

  2. Run docker-compose up -d

  3. See the ports and services running with docker ps

  4. Start requesting the status from the REST API by simply sending a GET request to http://0.0.0.0:8088/. It will return service information.

     {
     "KsqlServerInfo": {
         "version": "6.2.0",
         "kafkaClusterId": "uOXfee3zQ76vCKBbREw1yg",
         "ksqlServiceId": "default_",
         "serverStatus": "RUNNING"
     }
 }

Hope the strips some of you from the initial research. And.... if we are lucky there will be a wrapper library soon.

Then create a stream out of your topic and voila. You are ready to query your topic (through the stream) with the REST API. Since the REST API offers HTTP2, one could also expect continuous updates on freshly arriving records in the stream. Apply Push Queries for this. Pull queries will cut the line after the result has been delivered.

user_28152640
Related