How to use Kafka sink connector to database with partial record?

user_14342331 0
Viewed 16

I am having a kafka topic which receives the messages in this order

1 (emp id ; 321 , designation: abc , accountno : 1234)

now this record has been updated .The emp designation is changed, and the next message comes as below.

Only the changes are pushed other field are nullified.

2 (emp id :321 , designation:efg , accountno:null)

Is it possible to use the sink connector to replicate this into a database with correct updates?

End result is the database table should have record like this

emp :321 . designation :efg. accountno:1234
apache-kafka-connect apache-kafka
1 Answers

Each database sink connector works differently.

For example, the JDBC connector would store the null column(s); every field of the record is upserted into a row. The MongoDB connector may replace the whole document or allow partial document updates. The Elasticsearch sink works similarly with documents.

But in general, sure, as long as you have matching fields and some record key, an update operation should be possible, even if that first requires joining a partial record back into a full record using some external store.

user_23086830
Related