Does Kafka Streams library kill idle StreamThreads?

user_377762 0
Viewed 186

Say, KStream topology is simple: input topic -> process -> output topic. Partitions of input topic = 4.

If there is a single instance of app running with num.stream.threads=4, all 4 StreamThreads are utilized.

If a second instance is launched (with num.stream.threads=4), stream tasks are now distributed between the two. Task 0_1 and 0_2 on first instance, Task 0_3 and 0_4 on second instance.

On first instance, does kafka streams library kill the threads which were running 0_3 and 0_4 so far?

apache-kafka-streams apache-kafka
1 Answers

For your case when input topic has only 4 partitions, what will happen When starting 8 instances with num.stream.threads=1?

4 instances become idle but not be killed. They are remaining and get assign a task if any of other already assigned instance goes down.

So, same thing happens when you start multiple treads in one instances. In your case, 8 treads in 2 instances 4 per each. same scenario happen I will explained above. 4 of your threads getting idle and remain idle until it is getting a task by going down other instance.

more reference :

Let’s take an example. Imagine your application is reading from an input topic that has 5 partitions. How many app instances can we run here?

The short answer is that we can run up to 5 instances of this application, because the application’s maximum parallelism is 5. If we run more than 5 app instances, then the “excess” app instances will successfully launch but remain idle. If one of the busy instances goes down, one of the idle instances will resume the former’s work.

You can see more information of threads by set up metrics referring this

user_108667980
Related