Guarantee for fast data processing when scaling nodes up horizontal kafka

user_11254656 0
Viewed 46

I built a kafka application with java having:

  • a producer producing records to kafka
  • a kafka stream consuming these records, applying some (time windowed and with a state store) operations on its values and sending them back to kafka
  • a consumer consuming these transformed values and writing them into a database

I am measuring time differences between the kafka timestamps of the producer record (which gets consumed by the kafka stream) and the consumer record (which gets consumed by the consumer). So basically when the producer record gets created and when this record is transformed by the stream and sent back to kafka. In the end I take the average of each time difference in my database.

For any reason this time difference increases when I add more streaming nodes and more partitions to my topic. I actually expected the time difference to decrease. Now I am wondering if I am doing something wrong or if it can happen that by increasing the number of nodes the data processing takes a longer amount of time.

In the end my question is: Is it possible that the data processing takes longer by adding more nodes to kafka? If so, what could be the reasons?

apache-kafka-streams kafka-consumer-api apache-kafka
1 Answers

"Is it possible that the data processing takes longer by adding more nodes to kafka? If so, what could be the reasons?"

Yes, this could happen and is highly dependent on how much data is actually being produced. There needs to be a balance between amount of data and parallelism by using more partitions/stream nodes to avoid unnecessary overhead.

The main reason I can think of in your particular case is the batching on the KafkaProducer side is more efficient with a smaller amount of partitions.

Imagine you have 10 messages and one partition. The KafkaProducer will likely combine those 10 messages into one batch and apply a compression on it which seems quite efficient.

Now, if you have 10 messages and 10 partitions such that each message goes into its own partition, the KafkaProducer has to send 10 individual send requests to the broker (one for each partition), and also your compression rate is less efficient as you are always compressing only one single message.

In addition, if your KafkaProducer works in synchronous mode it has to wait more often on the reply of the broker (which can vary based on the Producer configuration acks and max.request.in.flight).

user_122089100
Related