I publish messages to a kafka topic (outputTopic) in a format which my subscribers can understand. I now wish to modify the format of these messages in a way which will break the existing topic consumers.
For example, I post objects, serialised in json format, but need to change the objects, and therefore the schema.
What is the best way to manage this type of change? Should I alter the producer so that it publishes to a new topic (outputTopic2)? Are there any better ways to manage this?
Avro schemas will not solve the problem. You could update the consumers to handle both old and new versions whether or not there are schemas.
Instead, keep your producer as it is. Deploy an updated version that reads from the same data source and publishes data to a new topic with the new updated format.
Allow consumers to migrate from the old version to the new version before finally killing the old one.
One clean way to do according to me, is to use Schema registry / Apache Avro. Depending on how you use it, it will help you to guarantee backward / forward compatibility.
