can we use kafka with nodejs without any external database?

Viewed 31

I am new to kafka and I want to use it with Node JS. I want to query the username and password of user in Node JS without external database. As for as I know Kstream and Ktables don't work with node JS.

If possible, can someone create me a simple REST API for verifying username and password? I can produce and consume messages in kafka but the main problem is that I can't integrate kafka when sending a request with to API with Node JS.

If we can't use it without an external database with nodejs, then why use it?

This is what I have tried:


        import avro from "avsc";
        import { Kafka } from "kafkajs";
    
    const Type = avro.Type.forSchema({
        type: "record",
        fields: [
            { name: "Kind", type: { type: "enum", symbols: ["CAT", "DOG"] } },
            { name: "name", type: "string" }
        ]
    })
    
    const buff = Type.toBuffer({ Kind: "CAT", name: "Arslan" })
    const val = Type.fromBuffer(buff);
    console.log(buff)
    console.log(val)
    
    const kafka = new Kafka({
        clientId: 'Producer-Application',
        brokers: ['localhost:9092', 'localhost:9093', 'localhost:9094'],
    })
    const ProducerFun = async () => {
    
        const producer = kafka.producer();
        await producer.connect();
        const MessageValueEncoded = await Type.toBuffer({ Kind: "CAT", name: "Arslan" })
        await producer.send({
            topic: 'test-topic1',
            messages: [{
                value: MessageValueEncoded
            }]
        })
        await producer.disconnect()
    }
    
    const ConsumerFun = async () => {
        const consumer = kafka.consumer({ groupId: "test-group-2" });
        await consumer.connect();
        await consumer.subscribe({ topic: 'test-topic1', fromBeginning: true });
    
        await consumer.run({
            eachMessage: async ({ topic, partition, message }) => {
                const val = await Type.fromBuffer(message.value);
                console.log(val)
            }
        })
    
    }
    ProducerFun().then(() => {
        ConsumerFun()
    })
1 Answers

Kafka doesn't need a database, and neither does NodeJS. If you need to use a database to store user information for indexed lookups, then that is an implementation detail of your own application, not a limitation of NodeJS / Kafka.

As mentioned in your previous question, there is an open source nodefluent/kafka-streams library that implements some KTable functions, or you can use ksqlDB to do similar operations and use its REST API.

Or you could embed SQLite into your own app, but you need to maintain state of that database if it is persisted on disk where you deploy your app. You'd also need to build your own way of handling distributed state since you'd have a unique SQLite database per-instance of your application (it cannot be scaled).

Related