What is the real difference between Task and SubTask in Flink

user_4443784 0
Viewed 280

I am confused with the concept of task and subTask in Flink.

If I have set an operator(like MapFunction)'s parallism to be 6, then, there would be 6 MapFunction instances in total, I think each instance is a subtask, I am not sure I have understood correctly(maybe we should say each instance is a task)

Task, from Flink source code'view, is a thread Runnable object, I would ask what would be run when a thread runs this runnable object, does it mean each operator instance(or with other operator instances because of operator chain) form a task?

apache-flink
2 Answers

This is unfortunately a bit fuzzy and is historically grown. If you have 6 MapFunctions, 6 tasks would be spawned according to the code-base, each running an operator instance (or more specifically a chain of operator instances).

However, conceptually, it's still only one task though (=a chain of operators). Subtask would on this level correspond to a chain of operator instances.

So you can see that it should be named subtask in the code. The documentation often tries to be more precise, but that generates a mismatch when you look into the code.

See also Difference between job, task and subtask in flink.

user_102993420

When you create a flink job it is actually a logical Query Execution Plan (QEP) and each operator is a task. When this QEP is deployed in the cluster it is called physical QEP and depending the parallelism X that you set it will have X sub tasks for each operator. Each subtask instance will be run in a thread, hence it is parallel.

Operator chain is possible only when the flow between the two subtasks are a simple forward. For instance, a map followed by a filter can be chained. But a keyBy followed by a reducer uses hash distribution in a called shuffle phase, in this case they cannot be chained.

So, if operators are chainned their subtasks of different phases are chainned and run by the same thread. But the subtasks parallel instances run in different threads.

user_20969860
Related