Kafka Stream Scala API slow performance

Viewed 951

I was exploring Kafka Streams API and implemented sample WordCountExample in Scala as well as in Java but when I run the Scala code it gives output very slow as compared to Java example.

Also I observed that when I produced record on wordCountInput topic it appears instantly in sinkWordCount topic but the result of wordCount is not displayed instantly in wordCountOutput topic if we run the Scala code but if we run the Java code then everything works fine and I can see the output in wordCountOutput topic instantly.

Java code:

public class WordCountExample {

        public static void main(String[] args) {
            Properties p = new Properties();
            p.put(StreamsConfig.APPLICATION_ID_CONFIG, "wordcount-application");
            p.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.1.93:9092");
            p.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
            p.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());

            StreamsBuilder builder = new StreamsBuilder();
            KStream<String, String> textLines = builder.stream("wordCountInput");
            KTable<String, String> wordCounts = textLines
                    .flatMapValues(textLine -> Arrays.asList(textLine.toLowerCase().split("\\+W")))
                    .groupBy((key, word) -> word)
                    .count()
                    .mapValues((key, value) -> value.toString());

            textLines.to("sinkWordCount");

            wordCounts.toStream().to("wordCountOutput");

            KafkaStreams streams = new KafkaStreams(builder.build(), p);
            streams.cleanUp();
            streams.start();

            Runtime.getRuntime().addShutdownHook(new Thread(() -> {
                streams.close();
            }));
        }
}

POM File of java program

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.mycompany</groupId>
    <artifactId>KafkaStreamExample</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <dependencies>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-streams</artifactId>
            <version>2.0.0</version>
        </dependency>
    </dependencies>
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
    </properties>
</project>

Scala code :

object KafkaStreamExample extends App {

  val props: Properties = {
    val p = new Properties()
    p.put(StreamsConfig.APPLICATION_ID_CONFIG, "wordcount-application")
    p.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.1.93:9092")
    p.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String.getClass)
    p.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String.getClass)
    p
  }
  val builder: StreamsBuilder = new StreamsBuilder()
  val sourceStream: KStream[String, String] = builder.stream[String, String]("wordCountInput")

  val wordCounts: KTable[String, String] = sourceStream
    .flatMapValues(textLine => textLine.toLowerCase.split("\\W+"))
    .groupBy((key, word) => word)
    .count()
    .mapValues((key, value) => value.toString)

  wordCounts.toStream.to("wordCountOutput")

  sourceStream.to("sinkWordCount")

  val stream: KafkaStreams = new KafkaStreams(builder.build(), props)

  sys.ShutdownHookThread {
    stream.close()
  }

  stream.cleanUp()
  stream.start()
}

Scala SBT :

name := "KafkaStreamExample"

version := "0.1"

scalaVersion := "2.12.6"

libraryDependencies += "org.apache.kafka" %% "kafka-streams-scala" % "2.0.0"
libraryDependencies += "javax.ws.rs" % "javax.ws.rs-api" % "2.1" artifacts( Artifact("javax.ws.rs-api", "jar", "jar"))

Kafka Version: kafka_2.11-2.0.0

so can any one tell me why is the scala example is not executing as fast as java example.

Update

Setting cache.max.bytes.buffering this property to 0 in Scala code gives the instant output of WordCount in Scala but this is not recommended to use in production according to this Stackoverflow answer also in java code we don't need to set this property but still the java give the fast output

0 Answers
Related