Set Table/Topic order in Apache Kafka JDBC

Question

Have 2 topics, source_topic.a , source_topic.b . source_topic.a have dependency with source_topic.b (eg. need to sink source_topic.b first). In order to note the sink process, need to sink data from source_topic.b first then sink from source_topic.a. Is there any way to set an order of topics / tables in source/sink configurations ?

Following are the configurations used and there are multiple tables and topics. The timestamp is used for the mode for updating a table each time it is polled. And timestamp.initial set value to a specific timestamp.

The Source Configuration

name=jdbc-mssql-prod-5
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
connection.url=jdbc:sqlserver:
connection.user=
connection.password=
topic.prefix= source_topic.
mode=timestamp
table.whitelist=A,B,C
timestamp.column.name=ModifiedDateTime

connection.backoff.ms=60000
connection.attempts=300

validate.non.null= false
# enter timestamp in milliseconds 
timestamp.initial= 1604977200000

The Sink Configuration

name=mysql-sink-prod-5
connector.class=io.confluent.connect.jdbc.JdbcSinkConnector
tasks.max=1
topics= sink_topic_a, sink_topic_b
connection.url=jdbc:mysql:
connection.user=
connection.password=

insert.mode=upsert
delete.enabled=true
pk.mode=record_key


errors.log.enable= true
errors.log.include.messages=true

user_350613 · Answer

No, the JDBC Sink connector doesn't support that kind of logic.

You're applying batch thinking to a streams world :) Consider: how would Kafka know that it had "finished" sinking topic_a? Streams are unbounded, so you'd end up having to say something like "if you don't receive any more messages in a given time window then assume that you've finished sinking data from this topic and move onto the next one".

You may be best doing the necessary join of the data within Kafka itself (e.g. with Kafka Streams or ksqlDB), and then writing the result back to a new Kafka topic which you then sink to your database.

Set Table/Topic order in Apache Kafka JDBC

1 Answers