Understanding the snowflake kafka connector config tuning parameters

user_458426 0
Viewed 143

I'm working on streaming ~2000 tables from Kafka to Snowflake using the Snowflake connector for the Kafka Connect platform. I would like to understand how to tune the parameters of the connector for the best throughput. Suggestions for Kafka and Kafka Connect settings are also welcome though my primary interest is understanding the connector parameters.

My topic sizes range from < 1GB to 100's of GB. We currently have only a single parition per topic and the topics are distributed across 30 connector tasks using the RoundRobin partitioner. Our max messages size across all topics is configured to 3MB and we are using AVRO with lz4 compression.

One of our largest topics has ~7 billion events on it and is only transferring to snowflake at a rate of ~2000 events/second. I imagine that increasing the number of partitions is my main lever but I also suspect 2000 events/second is lower than in could be with a change in configuration parameters.

The parameters I suspect should be tuned are:

  • buffer.count.records - default 10000 events
  • buffer.flush.time - default 120 seconds
  • buffer.size.bytes - default 5MB

Currently we are using the default values.

Any advice on how to use these parameters or others to increase our throughput?

apache-kafka-connect snowflake-cloud-data-platform
1 Answers

I can't speak from experience with the Kafka connector, but Snowflake prefers file in the size range of 10 - 100 MB, so I would expect that bigger that the default would be better, and also try a bigger warehouse.

user_20199780
Related