I want to load data from various S3 buckets (more than 10,000 buckets and each file is around 20-50MB) into Apache Kafka. The list of buckets is dynamic - buckets are added and removed at runtime. Ideally, each bucket configuration should have its own polling interval (how often to scan for new files - at-least 60 seconds, but might be much more) and priority (number of concurrent files being processed).
Note that setting up notifications from each of the S3 buckets to SQS/SNS/Lambda is not an option due to various IT policies in the organizations of each of the bucket owners.
Kafka Connect seems to be most commonly used tool for such tasks, and its pluggable architecture will make it easier to add new sources in the future, so it fits well. Configuring each S3 bucket as its own connector will let me set a different number of tasks (which maps to priorities) and polling interval for each one. And building a Java custom Kafka Connect source task for my expected file format sounds reasonable.
However, the Kafka Connect code indicates that each running task is assigned its own thread for the lifetime of the task. So if I have 10K buckets, each configured with its own connector and with a single task, I will have 10K threads running in my Kafka Connect distributed worker pool. That's a lot of threads that are mostly just sleep()-ing.
What is the correct approach to scaling the number of tasks/connectors in Kafka Connect?