Consider below scenario
Database Table1,Table2 -->(1)
--> (1) Debezium-Source-Connector --> Table1Topic, Table2Topic ----->(2)
--> (2) KStream (will convert and rename Table1Topic and Table2Topic to ProcessedTopic. basically kstream will process Table2Topic data only and add below fields without any id field in payload(updateByQuery, updateScriptSource,targetIndex) and Table1Topic will be just renamed to ProcessedTopic. Here Table2Topic have information which needed to be updated in target index in our case Table1Topic data is primary data with document id which we want to save in elastic search and Table2Topic data will be merged on top of Tale1Topic index by elasticsearch by _update_by_query)
updateByQuery <--- will be used to prepare query field for "_update_by_query" request
updateScriptSource <--- will contain update data for script source in "_update_by_query" request
targetIndex <--- contain target index name
Ex: { "script": { "source": updateScriptSource, "lang": "painless" }, "query": updateByQuery }
Note: Kstream is my personal implementation to process topics
(2)---> ProcessedTopic ---> (3)ElasticSearch-Sink-Connector --->(4) ElasticSearch
I want to achieve above scenario. If there is already some similar implementation available then could someone please direct me. Any help is appreciated.
Thanks