Kafka to Snowflake connection issue

user_449355 0
Viewed 807

I am trying to connect from local standalone Confluent Kafka topics to Snowflake tables. I am using the following connector config via ksqldb.

CREATE SINK CONNECTOR `snowflake_sink` WITH(
"name"='snowflake_sink',
"tasks.max"='1',
"connector.class"='com.snowflake.kafka.connector.SnowflakeSinkConnector',
"topics"='USERPROFILE',
"snowflake.url.name"='https://mybu.mycompany.us-east-1.aws.privatelink.snowflakecomputing.com',
"snowflake.user.name”=‘myuser’,
"snowflake.database.name"='DEMO_DB',
"snowflake.topic2table.map"='USERPROFILE:UK_SF_DB1_Table1',
"snowflake.schema.name"='PUBLIC',
"snowflake.private.key”=‘<valid private key>’,
"snowflake.private.key.passphrase”=‘<valid pass phrase>’,
"key.converter"='org.apache.kafka.connect.storage.StringConverter',
"value.converter"='com.snowflake.kafka.connector.records.SnowflakeJsonConverter',
"key.converter.schema.registry.url"='http://schema-registry:8081',
"value.converter.schema.registry.url"='http://schema-registry:8081');

Ksqldb throws error saying snowflake url, username, private key is not valid. They are all valid as I can login using the same via Snowflake web ui.

ksqldb snowflake-cloud-data-platform apache-kafka
2 Answers

Some ideas on troubleshooting this:

  1. it looks like you have some "smart quotes" in your config, please remove the smart single and double quotes and replace them with proper text quotes. This may have happened during your preparing of this question, so just double check them.

  2. set your table name to all capital letters.

  3. your snowflake account should not include the https:// prefix. Examples at these links:

https://community.snowflake.com/s/article/Docker-Compose-Setting-up-Kafka-using-Snowflake-Sink-Connector-and-testing-Streams

https://docs.snowflake.com/en/user-guide/kafka-connector-install.html

user_87675700

You should remove the smart quotes and use the normal one like 'snowflake.url.name'='my.domain.com:443',

user_51488090
Related