Publishing whole file as a single message in Kafka topic using Java

user_8903523 0
Viewed 479

I have a use case where I need to publish whole file as a single message using Kafka Producer, so that when the consumer reads the message, consumer should be able to able to get whole file. I tried implementing this by converting the file to byte array and publish it to topic

byte[] fileData = Files.readAllBytes(Paths.get(path));
producer.send(new ProducerRecord<String, byte[]>("test", fileData));

When the same is consumed, each line is coming as a separate message

kafka-producer-api apache-kafka
1 Answers

Kafka has a default record size of 1MB, and is not meant for file delivery.

The recommended pattern is to

  1. Upload your file to a shared filesystem (such as S3)
  2. Take the path to the file as a URI (such as s3://path/to/file)
  3. Produce that URI in the topic record
  4. Consume
  5. Download file from filesystem

Without seeing your consumer, it's hard to diagnose your exact issue with the consumer, but you are reading the file fine.

user_23086830
Related