Kafka Message Chunking

user_15710678 0
Viewed 53

I am very new to Apache Kafka and I want my messages to be splitted into chunks when sent to the consumer. When the message bytes larger than the maximum request size, I want them to be splitted into chunks.

So I implemented it, however I want to guarantee that the messages are sent in the same order. How can I achieve this? If you want to review my code, here is the link: https://github.com/beyzayildirim137/msgChunking.git

Hope someone can help.

apache-kafka
1 Answers

Ideally Kafka is designed for smaller messages, but if you want to use it to stream large messages then the below strategy can be used (Note: there can be other strategies based on the use case):

  • Break the message into smaller chunks
  • Make sure all the chunks of a given message is published to the same partition, this will ensure the ordering (From your code on github looks like you have already done using same UUID as key for the all the chunks of a given message)
  • On the last chunk add some metadata in the header representing the end of the message while producing so that consumer can know this is the last chunk and it can reconstruct the message. (From the github code, the consumer code seems flaky as you are assuming all the messages coming in the 1000ms polling window will contain all the chunks which may not be the case)

I hope this gives you some direction to handle larger messages.

user_16819740
Related