Kafka JDBC Sink handle array datatype

user_14560049 0
Viewed 775

I know that Kafka JDBC Sink Connector have some drawbacks for array datatype. However is it possible to combine the Sink Connector with a simple Kafka Connector which can support array datatype. How can I filter from Kafka configuration and switch them into simple Kafka Connector configuration What does simple Kafka Configuration mean? How can Kafka Connect support array fields

name: topic_name
type: array
item: Topic file

Is this possible where it will consume to the db as a string not an array

"fields":[{
  "name":"item_id",
  "type":{
     "type":"array",
     "items":["null", "string"]
  },
"default":[]
}]
}
apache-kafka-connect apache-kafka avro jdbc
1 Answers

Kafka Connect framework itself doesn't expose limitations around types, it's in the source code for the JDBC sink that arrays are rejected.

There's an outstanding PR to support it for Postgres - https://github.com/confluentinc/kafka-connect-jdbc/pull/805

Unclear what you mean by "simple", but if you want to use a different connector, then you'd need to install it, and then change the class. For example, maybe the MongoDB sink handles arrays. I know that S3 and HDFS sinks do...

is it possible to combine the Sink Connector with a simple Kafka Connector

Again, unsure what you mean by this, but connectors generally don't "chain together". While you could use MirrorMaker2 with a transform to effectively do the same work as Kafka Streams, best to use the more appropriate tools for that

Is this possible where it will consume to the db as a string not an array

Sure, if the message field is actually a string. As suggested, you're going to need to process the message before the sink connector consumes it

user_23086830
Related