I know a task manager could have several task slots.
But, what is a task slot ? A JVM process or an object in memory or a thread?
5 Answers
The answer might come late. But:
A Taskmanager (TM) is a JVM process, whereas a Taskslot (TS) is a Thread within the respective JVM process (TM). The managed memory of a TM is equally split up between the TS within a TM. No CPU isolation happens between the slots, just the managed memory is divided. Moreover, TS in the same TM share TCP connections (via multiplexing) and heartbeat messages. They may also share data sets and data structures, thus reducing the per-task overhead.
Each worker (TaskManager) is a JVM process, and may execute one or more subtasks in separate threads. To control how many tasks a worker accepts, a worker has so called task slots (at least one).
@Janukowitsch's answer is good. Additionally, a TS represents a fixed subset of resources of the TM. One TS can take more than 1 thread. Actually one subtask/task is executed by one thread. Multiple subtasks/tasks can be deployed in the same slot.
I would like to address some inconsistencies in the previous answers:
TaskSlot is a Thread
With default settings TaskSlot != Thread. TaskSlot = Thread only if slot sharing is disabled. Slot sharing is an optimization that is on by default and, in most cases, you would want to keep it that way. It is more precise to say that each Operator Chain gets a separate Thread [1] spawn for its execution.
a TS represents a fixed subset of resources of the TM
This is not the case. A Task Slot is purely a scheduling concept that gives some control over how many Operators or Operator Chains will be placed on which task manager. There is no guaranteed allocation of a fixed set of resources to a Task Slot. For instance, if you have a Task Manager with 5 cores and 5GB of RAM which is assigned two Task Slots, those slots will be "competing" for both the CPU and the memory - they are not split in any fixed manner.
The number of slots is typically proportional to the number of available CPU cores of each TaskManager. As a general recommendation, the number of available CPU cores is a good default for taskmanager.numberOfTaskSlots.
Related