Kafka JDBC Sink Connector can't find tables in Snowflake

user_657468 0
Viewed 607

I'm trying to use the JDBC Sink Connector to write data into Snowflake but I keep getting the error below suggesting that the table can't be found in the database (I have auto creation disabled since it's not supported with Snowflake).

[2021-11-08 18:52:19,300] INFO [test-connector-jdbc|task-0] Checking Generic dialect for existence of TABLE "test_database"."test_schema"."test_table" (io.confluent.connect.jdbc.dialect.GenericDatabaseDialect:575)
[2021-11-08 18:52:19,455] INFO [test-connector-jdbc|task-0] Using Generic dialect TABLE "test_database"."test_schema"."test_table" absent (io.confluent.connect.jdbc.dialect.GenericDatabaseDialect:583)
[2021-11-08 18:52:19,572] ERROR [test-connector-jdbc|task-0] Error encountered in task test-connector-jdbc-0. Executing stage 'TASK_PUT' with class 'org.apache.kafka.connect.sink.SinkTask'. (org.apache.kafka.connect.runtime.errors.LogReporter:66)
io.confluent.connect.jdbc.sink.TableAlterOrCreateException: Table "test_database"."test_schema"."test_table" is missing and auto-creation is disabled
at io.confluent.connect.jdbc.sink.DbStructure.create(DbStructure.java:118)
at io.confluent.connect.jdbc.sink.DbStructure.createOrAmendIfNecessary(DbStructure.java:67)
at io.confluent.connect.jdbc.sink.BufferedRecords.add(BufferedRecords.java:123)
at io.confluent.connect.jdbc.sink.JdbcDbWriter.write(JdbcDbWriter.java:74)
at io.confluent.connect.jdbc.sink.JdbcSinkTask.unrollAndRetry(JdbcSinkTask.java:133)
at io.confluent.connect.jdbc.sink.JdbcSinkTask.put(JdbcSinkTask.java:87)
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:581)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:329)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:232)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:201)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:186)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:241)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

I can see the following query being run from the JDBC driver in Snowflake (and running it directly in Snowflake, with the same user configured in the JDBC connector, does return data.

show /* JDBC:DatabaseMetaData.getTables() */ tables like 'TEST_TABLE' in account

And finally my configuration below:

{
    "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
    "tasks.max": "1",
    "connection.url": "jdbc:snowflake://######.eu-west-1.snowflakecomputing.com/?db=test_database&schema=test_schema&role=role",
    "connection.user": "test_user",
    "connection.password": "test_password",
    "topics.regex": "test_table",
    "table.name.format": "test_database.test_schema.test_table_${topic}",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "errors.tolerance": "all",
    "errors.deadletterqueue.topic.name": "dlq",
    "errors.deadletterqueue.topic.replication.factor":"1",
    "errors.log.enable":"true",
    "quote.sql.identifiers":"never"

}

Any idea what I might be missing here?

apache-kafka-connect snowflake-cloud-data-platform apache-kafka jdbc
2 Answers

Turns out this was caused by the Kafka Connect JDBC Connector using getTables() which is case sensitive, as highlighted in this answer: https://stackoverflow.com/a/62232384/657468

The "quote.sql.identifiers":"never" parameter only applies when the connector attempts to insert, not when looking for existing tables in the database.

user_6574680

I was able to get this working with auto creation and SMT's using the JDBC sink.

I downloaded the Snowflake JDBC driver and add the jar to the connectors class path in your Kafka instance. I ran into an issue on my Kafka instance complaining there was no suitable JDBC driver in Snowflake.

Here's my configuration:

connector.class=io.confluent.connect.jdbc.JdbcSinkConnector
transforms.unwrap.delete.handling.mode=rewrite
transforms.InsertField.timestamp.field=rowtime
tasks.max=1
timezone=utc
transforms=unwrap,InsertField
value.converter.enhanced.avro.schema.support=true
transforms.InsertField.type=org.apache.kafka.connect.transforms.InsertField$Value
auto.evolve=true
transforms.unwrap.drop.tombstones=true
transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
value.converter=io.confluent.connect.avro.AvroConverter
insert.mode=upsert
key.converter=io.confluent.connect.avro.AvroConverter
dialect.name=OracleDatabaseDialect
topics=<test_table>
key.converter.enhanced.avro.schema.support=true
value.converter.schema.registry.url=http://localhost:8081
auto.create=true
connection.url=jdbc:snowflake://<snowflake_account>.snowflakecomputing.com/?user=<user>&password=<secret_password>&warehouse=demo_wh&db=demo_db&schema=public&role=sysadmin
pk.mode=record_key
key.converter.schema.registry.url=http://localhost:8081
pk.fields=SquareId
user_109704780
Related