When we select any table in Flink SQL CLI which is a Source to the SQL Server Database table, why does the Flink job "Finishes" after it fetches all the records from the database table? Is it possible to make it "Running" so it can fetch the latest data from database tables?
1 Answers
The JdbcTableSource in JDBC connector is defined as a bounded source which means it will finish after fetched all records rather than monitoring the SQL Server Database table changing and then send database table changelog continuously.
Maybe flink CDC connector is what you want, it supports reading database table snapshot and continues to read table binlogs. Flink also supports CDC format with Kafka(e.g: a kafa topic which contains table binglog which loading by CDC tools), you can continuous read the lates table from Kafka table that defined in Flink SQL.
Flink works well with CDC connector and CDC format, you can lookup [1][2] for more information.
[1] flink-cdc-connector: https://github.com/ververica/flink-cdc-connectors
[2] flink CDC format: https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/connectors/formats/debezium.html
Related