I trying to run a Kafka Connect distributed cluster with 5 workers and 2 sink connectors. One connector is configured with max.tasks and actual tasks set to 5. The other connector is configured for 6 tasks.
If I understand things correctly this should mean 13 "work units" (1 connector + 5 tasks + 1 connector + 6 tasks) that need to be distributed to the workers. Which i figured would mean around 2-3 tasks/connectors per worker.
However, I'm finding that the eventual assignments are as follows:
Worker 0: (3 Work Units)
- Connector 0
- Connector 1 Task 0
- Connector 1 Task 1
Worker 1: ( 4 Work units)
- Connector 1
- Connector 0 Task 0
- Connector 0 Task 2
- Connector 1 Task 3
Worker 2: (2 Work units)
- Connector 0 Task 3
- Connector 0 Task 4
Worker 3: (4 Work units)
- Connector 0 Task 1
- Connector 1 Task 2
- Connector 1 Task 4
- Connector 1 Task 5
Worker 4: ( 0 Work Units)
I'm using Kafka Connect workers running 2.6.0 and a connector built against 2.6.0 libraries. Workers are deployed as containers in an orchestration system (Nomad) and have gone through several rounds of rolling restarts/reallocations but the balance always seems off.
Based on this I'm finding myself at a loss for the following:
- I'd get that tasks may get a bit unbalanced while restarting or moving containers but i'd expect once the last one came up and joined the cluster a final balancing would have sorted things out. Can some one point me to why this might not be happening?
- Is there a recommended way to trigger a rebalancing for the whole cluster? From the docs it seems that changing config for a connector or a worker failing/joining might cause a rebalance but that does not seem to be an ideal process.
- Can the task balancing process be configured or controlled in anyway?