Does using JSON serializer in kafka streams cause a performance hit compared to AVRO?

user_2655966 0
Viewed 3941

I am using the following configuration for JSON in my Kafka Streams app,

properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
            Serdes.String().getClass());
properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
            GenericJSONSerde.class);
properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
            com.buy.json.JsonSerializer.class);
properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
            com.buy.json.JsonDeserializer.class);
properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
            com.buy.json.JsonSerializer.class);

To measure the performance, I removed all Logic and what ever I receive from IN topic, I am just pushing it to OUT topic. It takes 3 minutes to process 350,000 records

But when I use AVRO, the same streams app process 3 Million records in just 1 Minute. Here is my AVRO configuration,

  `properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
            Serdes.String().getClass());
    properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
            GenericAvroSerde.class);
    properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
            io.confluent.kafka.serializers.KafkaAvroSerializer.class);
    properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
            io.confluent.kafka.serializers.KafkaAvroSerializer.class);
    properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
            io.confluent.kafka.serializers.KafkaAvroDeserializer.class);
    properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
            io.confluent.kafka.serializers.KafkaAvroDeserializer.class);`

The streams app that I am writing should support Both AVRO/JSON. I will decide that while starting the streams app.

My question is: IS using JSON a performance hit in Streams app? Please clarify.

apache-kafka-streams apache-kafka json
2 Answers

Note : In case of Confluent's Schema Registry the schema is sent over HTTP first as discussed in comments.

AVRO is obviously fast because it provide a compact binary data format. There is a concept of dynamic typing and tagging of data during serialization .

In case of AVRO ,the data and schema ( in form of JSON ) is always together ,hence the size become less for data type info during deserialization, better handling of static data types and less code generation.

This is not present in case of JSONSerializer .

user_24425650

Compared to AVRO, JSON might be slower in general, because JSON is a text-based format whereas AVRO is a binary format. This is independent of Kafka Streams.

The size of data encoded in JSON is generally larger, which impacts network transmission throughput. AVRO might generally be de-/serialized faster than JSON.

See the following comparison from 2017 for more information: https://labs.criteo.com/2017/05/serialization/

user_68222440
Related