how to write same message in all partitions of a single kafka topic?

user_11068674 0
Viewed 3634

I have a single Topic suppose name "Test". Suppose it has 4 partition P1, P2, P3, P4. Now, I am sending a message to suppose M1 from Kafka Producer. I want message M1 to get written in all partition P1, P2, P3, P4. Is it Possible? If yes then how I can do that? (I am new to this, I am using Kafka-Node to do this.)

kafka-producer-api kafka-consumer-api apache-kafka
1 Answers

According the to documentation on a ProducerRecord you can specify the partition of a ProducerRecord. That way you can write the same message to multiple partitions of the same topic. The api for this look like this in Java:

ProducerRecord(String topic, Integer partition, K key, V value)

Overall your approach could look like this, although I am also questioning this approach of duplicating data and would rather re-consider a design change.

Producer<String, String> producer = new KafkaProducer<>(props);
 for (int part = 0; part < 4; part++)
     producer.send(new ProducerRecord<String, String>("Test", part, "Hello", "World!"));

 producer.close();

EDIT (after comment from OP with more background on use case):

From your comment I understand that you want to read the data in parallel and perform two different steps. Instead of writing the same message to two different partitions within the same topic I'd rather recommend to have the data stored only once in your topic (meaning in any partition). On the consumer side, you can make sure that your 2 consumer have a different ConsumerGroup (configuration: group.id). If they have two different ConsumerGroups they will be able to process the data in parallel. Kafka will not drop the message if it has been consumed, so it can be consumed by as many different(!) ConsumerGroups as you like. Data in Kafka is only deleted based on retention times or size which is configured at topic level and is independent of Producer/Consumer.

user_122089100
Related