Apache Camel Kafka Connector: write to GCS in Avro format

user_219449 0
Viewed 137

I would like to consume events from Kafka which are in Protobuf and write them to GCS with Apache Camel Kafka Connector. I use Protobuf converter so events have both value and schema attached.
Would it be possible for me to write those events to Google Cloud Storage in Avro format (GenericDatumWriter with schema)?
I would also like to batch events into bigger files.

I could not find examples of writing files in Avro, any guidance would be appreciated!
Cheers!

apache-kafka-connect apache-camel
1 Answers
  1. To use the official Confluent ProtobufConverter, a Confluent schema registry is needed. This official ProtobufConverter is able to convert bytes that are serialised by probobuf.
  2. Consider using Kafka Connect GCS connector.
    • The first question is: why Apache Camel connector is chosen?
    • toConnectData method in converters returns SchemaAndValue which can be converted to Avro/Parquet/JSON schema and data. This is the very reason why GCS connector is able to support multiple output formats.
user_13974730
Related