Tuning kafka streams for speed

Question

I have two streams:

[topicA] -> processingA -> [topicB] -> processingB -> [topicC]

With logging in my application I have noticed that between sending output from processingA to topicB and picking message from topicB for processingB it takes more than 100ms (rather 150ms) each time. It might not be much, but it accumulates and finally rather simple cascade of processing takes almost second.

Can I tune kafka so that these delays are as close to zero as possible? What config params have impact on these delays?

I have mostly default config. Is it commit.interval.ms that causes delay? I already changed it from much higher default value...

StreamsConfig values: 
    application.id = app
    application.server = 
    bootstrap.servers = [localhost:9092]
    buffered.records.per.partition = 1000
    cache.max.bytes.buffering = 10485760
    client.id = 
    commit.interval.ms = 100
    connections.max.idle.ms = 540000
    default.deserialization.exception.handler = class org.apache.kafka.streams.errors.LogAndContinueExceptionHandler
    default.key.serde = class org.apache.kafka.common.serialization.Serdes$StringSerde
    default.production.exception.handler = class org.apache.kafka.streams.errors.DefaultProductionExceptionHandler
    default.timestamp.extractor = class org.apache.kafka.streams.processor.FailOnInvalidTimestamp
    default.value.serde = class org.apache.kafka.common.serialization.Serdes$StringSerde
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    num.standby.replicas = 0
    num.stream.threads = 1
    partition.grouper = class org.apache.kafka.streams.processor.DefaultPartitionGrouper
    poll.ms = 100
    processing.guarantee = exactly_once
    receive.buffer.bytes = 32768
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    replication.factor = 1
    request.timeout.ms = 40000
    retries = 0
    retry.backoff.ms = 100
    rocksdb.config.setter = null
    security.protocol = PLAINTEXT
    send.buffer.bytes = 131072
    state.cleanup.delay.ms = 600000
    state.dir = /tmp/kafka-streams
    topology.optimization = none
    upgrade.from = null
    windowstore.changelog.additional.retention.ms = 86400000

pgras_12719 · Answer

The 100 to 150ms seems normal in your situation as you have set commit.interval.ms = 100 (which is also the default for processing.guarantee = exactly_once). With exactly once processingB will only read messages that have been commited in [topicB], and processingA will only commit after (at least) 100ms.

Without re-architecting your system, you could tune commit.interval.ms for a better latency, but if you lower that value, the throughput will also be lower.

See this post (Streams Performance Implications) which describes this trade-of.

user_96180 · Answer

what is your consumer/processingA polling frequency ?

try to reduce that so that your processor will consume the data quickly

and also your poll.ms is set to 100ms reduce that as well and see if that helps.

Tuning kafka streams for speed

2 Answers