Kafka: Consumer API vs Streams API

user_2582966 0
Viewed 55765

I recently started learning Kafka and end up with these questions.

  1. What is the difference between Consumer and Stream? For me, if any tool/application consume messages from Kafka is a consumer in the Kafka world.

  2. How Stream is different as this also consumes from or produce messages to Kafka? and why is it needed as we can write our own consumer application using Consumer API and process them as needed or send them to Spark from the consumer application?

I did Google on this, but did not get any good answers for this. Sorry if this question is too trivial.

apache-kafka-streams kafka-consumer-api apache-kafka
3 Answers

Kafka Stream component built to support the ETL type of message transformation. Means to input stream from the topic, transform and output to other topics. It supports real-time processing and at the same time supports advanced analytic features such as aggregation, windowing, join, etc.

"Kafka Streams simplifies application development by building on the Kafka producer and consumer libraries and leveraging the native capabilities of Kafka to offer data parallelism, distributed coordination, fault tolerance, and operational simplicity."

Below are key architectural features on Kafka Stream. Please refer here

  1. Stream Partitions and Tasks: Kafka Streams uses the concepts of partitions and tasks as logical units of its parallelism model based on Kafka topic partitions.
  2. Threading Model: Kafka Streams allows the user to configure the number of threads that the library can use to parallelize processing within an application instance.
  3. Local State Stores: Kafka Streams provides so-called state stores, which can be used by stream processing applications to store and query data, which is an important capability when implementing stateful operations
  4. Fault Tolerance: Kafka Streams builds on fault-tolerance capabilities integrated natively within Kafka. Kafka partitions are highly available and replicated, so when stream data is persisted to Kafka it is available even if the application fails and needs to be re-process.

Based on my understanding below are key differences I am open to updates if missing or misleading any point

enter image description here enter image description here

Where to use Consumer - Producer:

  1. If there are single consumers, consume the message process but not spill to other topics.
  2. As point 1 if having just a producer producing message we don't need Kafka Stream.
  3. If consumer messages from one Kafka cluster but publish to different Kafka cluster topics. In that case, you can even use Kafka Stream but have to use a separate Producer to publish messages to different clusters. Or simply use Kafka Consumer-Producer mechanism.
  4. Batch processing - if there is a requirement to collect a message or kind of batch processing it's good to use a normal traditional way.
  5. If you are looking for more control over when to manual commit

Where to use Kafka Stream:

  1. If you consume messages from one topic, transform and publish to other topics Kafka Stream is best suited.
  2. Real-time processing, real-time analytics, and Machine learning.
  3. Stateful transformation such as aggregation, join window, etc.
  4. Planning to use local state stores or mounted state stores such as Portworx etc.
  5. Achieve Exactly one processing semantic and auto-defined fault tolerance.
user_25318430

Streams builds upon the Consumer and Producer APIs and thus works on a higher level, meaning

  • Streams is easier to use for read-from-topic/process/write-to-topic style tasks
  • Producer/Consumer allows for more control and can be used in some cases that Streams does not handle

As an example, Streams handles transaction commits automatically, which means you cannot control the exact point in time when to commit, (regardless of whether you use the Streams DSL or the Processer API). The Consumer/Producer API in contrast gives you that control.

user_7095370
Related