Read Avro messages from Kafka in terminal - kafka-avro-console-consumer alternative

user_1683626 0
Viewed 1094

I'm trying to find easiest way how to read Avro messages from the Kafka topics in readable format. There is option to use Confluent kafka-avro-console-consumer in following way

./kafka-avro-console-consumer \
   --topic topic \
   --from-beginning \
   --bootstrap-server bootstrap_server_url \
   --max-messages 10 \
   --property schema.registry.url=schema_registry_url

but for this I need to download whole Confluent platform (1.7 GB) that I see as an overkill in my scenario.

Is there any alternative how I could get Avro messages from the Kafka topics in the terminal easily?

confluent-platform apache-kafka avro
3 Answers

You will need to download additional Kafka tools that supports the Schema Registry and Avro format, such as ksqlDB or Conduktor or AKHQ or similar GUI tools

kcat might support Avro now, I cannot recall

You could write your own consumer script. The Python library from Confluent doesn't require much code to consume Avro records

You could also clone the Schema Registry project from Github and build it on its own, then use the CLI scripts there

user_23086830

I was able to get the last Avro messages in the readable form with kcat

kcat -C -b bootstrap_server \
    -t topic \
    -r schema_registry \
    -p 0 -o -1 -s value=avro -e
user_16836260

Another way to read messages in Avro is to use Kafdrop. Test it by adding such section to your docker-compose.yml along with broker, server-regestry and other containers:

  kafdrop:
    image: obsidiandynamics/kafdrop
    restart: "no"
    ports:
      - "9001:9000"
    environment:
      KAFKA_BROKERCONNECT: "broker:9092"
      JVM_OPTS: "-Xms16M -Xmx48M -Xss180K -XX:-TieredCompilation -XX:+UseStringDeduplication -noverify"
      CMD_ARGS: "--message.format=AVRO --schemaregistry.connect=http://schema-registry:8081"
    depends_on:
      - "broker"

After that, open Kafdrop at http://localhost:9001, click at topic where avro-messages will be put, and choose AVRO message format in drop down.

user_45748000
Related