How do you use a nodejs Lambda in AWS as a producer to send messages to MSK topic without creating EC2 client server?

Viewed 2183

I am trying to create a Lambda in AWS that serves as a producer to an MSK topic. All the AWS docs say to create a new EC2 instance, but as my Lambda is in the same VPC I feel like this should work. I am very new to this and I notice my log statement never hits in my producer.on function. I am using nodejs and the kafka-node module. The code can be found below.

Essentially, I am just wondering if anyone knows how to do this and why the producer.on function is never hit when I run test through the Lambda? This is just some test code to see if I can get it to send, but if any more data is needed oy help please let me know and thanks in advance.

exports.handler = async (event, context,callback) => {

const kafka = require('kafka-node');
const bp = require('body-parser');

const kafka_topic = 'MyTopic';
const Producer = kafka.Producer;
var KeyedMessage = kafka.KeyedMessage;
const Client = kafka.Client;
const client = new kafka.KafkaClient({kafkaHost: 'myhost:9094'});

console.log('client :: '+JSON.stringify(client));

const producer = new Producer(client);
console.log('about to hit producer code');

producer.on('ready', function() {
  console.log('Hello there!')
  let message = 'my message';
  let keyedMessage = new KeyedMessage('keyed', 'me keyed message');
  
    producer.send([
    { topic: kafka_topic, partition: 0, messages: [message, keyedMessage], attributes: 0 }
  ], function (err, result) {
    console.log(err || result);
    process.exit();
  });
});

producer.on('error', function (err) {
  console.log('error', err);
});
}
return "success";
1 Answers

What you need is to be able to produce messages on your MSK cluster using REST API. Why not setup a REST proxy for MSK as detailed here and then call this API to produce your messages to MSK.

Related