Pitfalls of kafka tiered storage

user_8906627 0
Viewed 665

I have a question regarding the tiered storage feature in Kafka. I like this feature since it means in my case that I can use Kafka as infinite storage (with gcs backend for example). However, let's suppose for whatever reason the Kafka cluster got deleted and Kafka data is lost.

Is data in gcs/s3 store still useful?

I mean can I plug the old logs to a new Kafka cluster or is it totally useless now (terabytes of logs)?

BTW I know I can analyse the segments in the gcs/S3 store and extract data. but that's a bit hacky that's why I m trying to see if I can find a clean solution.

confluent-platform apache-kafka
1 Answers

As of right now, if the cluster or specifically the topic that has tiered storage enabled gets deleted, the data in GCS/S3 will not be "reloaded" if you connect it to another cluster.

If you want to keep the data that's in GCS/S3, you will need to stream the data to a new topic that does not have tiered storage enabled or use kafka connect to independently write the data to a usable format before deleting it.

We do plan on improving this use case in the future.

user_32727070
Related