Apache Flink job is not scheduled on multiple TaskManagers in Kubernetes (replicas)

user_1353364 0
Viewed 539

I have a simple Flink job that reads from an ActiveMQ source & sink to a database & print. I deploy the job in the Kubernetes with 2 TaskManagers, each having Task Slots of 10 (taskmanager.numberOfTaskSlots: 10). I configured parallelism more than the total TaskSlots available (ie., 10 in this case).

When I see the Flink Dashboard I see this job runs only in one of the TaskManager, but the other TaskManager has no Jobs. I verified this by checking every operator where it is scheduled, also in Task Manager UI page one of the manager has all slots free. I attach below images for reference.

Did I configure anything wrong? Where is the gap in my understanding? And can someone explain it?

enter image description here

Jobenter image description here

enter image description here

flink-cep flink-streaming apache-flink kubernetes
1 Answers

The first task manager has enough slots (10) to fully satisfy the requirements of your job.

The scheduler's default behavior is to fully utilize one task manager's slots before using slots from another task manager. If instead you would prefer that Flink spread out the workload across all available task managers, set cluster.evenly-spread-out-slots: true in flink-conf.yaml. (This option was added in Flink 1.10 to recreate a scheduling behavior similar to what was the default before Flink 1.5.)

user_20008230
Related