retention.ms not set on changelog topics created by WindowsStoreBuilder

user_972912 0
Viewed 464

Kafka stream automatically set retention.ms and cleanup.policy on internal topics, ex. materialized KTables. However, I observed that retention.ms isn't set on logging topics created by WindowStoreBuilder.

Intuitively, I had say that logging topics are also internal topics and could / should be auto-configured. In my case, store has very short retention few minutes but default retention.ms is 5 days. Restoring a local store takes one hour while we only care about the last few minutes of data and older data will be evicted from the store.

Am I doing something wrong or should I explicitly set those config entries?

        var retention = Duration.ofMinutes( 5 );
        var storeBuilder =
                new WindowStoreBuilder<>(
                        Stores.persistentWindowStore(
                                "name",
                                retention,
                                Duration.ofMinutes( 1 ),
                                false ),
                        Serdes.String( ), new JaegerSpanSerde( ), Time.SYSTEM )
                                .withLoggingEnabled( Map.of(
                                        TopicConfig.RETENTION_MS_CONFIG, Long.toString( retention.toMillis( ) ),
                                        TopicConfig.SEGMENT_MS_CONFIG, Long.toString( retention.toMillis( ) ) ) );

If retention is increased in a subsequent release and changelog topic already exist, will topic configuration be updated?

apache-kafka-streams
1 Answers

Internal topics are only configures by Kafka Streams when they are creates. If you change your retention time setting in your code, it won't update the corresponding topic config. This is a known issue: https://issues.apache.org/jira/browse/KAFKA-7591

As a workaround, you can manually reconfigure the changelog topic.

user_49530790
Related