What is the relationship between a Node, Worker, Executor, Task and Partition

user_3256598 0
Viewed 610

I am trying to understand the relationship between different components and elements in the Spark architecture but am unable to get a grip on it. Can someone please validate my assumptions and correct me where I am wrong.

  1. My understanding - A node is the actual physical machine. One node can contain the main driver while others will contain the workers.
    Q - Can a node have multiple drivers (if I have multiple applications)?

  2. My understanding - A worker is a process within a node. There can be multiple workers within each node though not recommended.

  3. My understanding - An executor is a sub-process(?) within a worker process. Each worker can have multiple executors.
    Q. What metric determines the number of executors per worker?
    Q. Is the idea of a JVM associated with an executor process or at a higher "worker" level?
    Q. What is the relationship between a core and an executor?
    Q - Can RAM and HD be allocated at an executor level?
    For e.g., if I have a worker node with 100GB of RAM and 5 TB HD, can I allocate 20 GB RAM and 1 TB HDD per executor for that worker?

  4. My understanding - A partition is a portion of the actual data. This split could happen using hashing, round robin or range.
    Q - What determines the location of these data partitions?
    For e.g., if I have a cluster with 2 nodes, 10 executors (5 executors in each node) and a dataframe with 20 partitions, I'm assuming I would have 2 partitions in each executor or is there a chance that partition distribution could be skewed? What would I need to do to ensure that all my partitions that have a certain partitioning key get co-located within the same worker so there is minimum network transfer when these partitions have to work together to, say, perform an aggregation or a join?
    Q - What happens when a repartition() is performed. For e.g., if I have 20 partitions across 10 executors (say, 2 partitions in each) and I repartition(2). I will now have only 2 portions of data which I assume would be resting in a couple of executors. What happens to the remaining executors?

  5. Assumption - A task is the lowest unit of work that performs the actual ask. The number of tasks dependent on the number of partitions. So, if there are 20 partitions, I would have 20 tasks in each stage.
    Q - Are these tasks performed by individual executors?
    Q - If I have less executors (say, 10) than the partitions (say, 20), does it mean that only 10 tasks will be executed in parallel at any point? is the degree of parallelism constrained by the number of executors?

Thanks in advance!

apache-spark
1 Answers
  1. My understanding - A node is the actual physical machine. One node can contain the main driver while others will contain the workers. (This is Correct as a starting Point)

Q - Can a node have multiple drivers (if I have multiple applications)?

Yes Because driver is just a process that gets created based on the program that you might have written. And you can have multiple process running on the same node.

  1. My understanding - A worker is a process within a node. There can be multiple workers within each node though not recommended.

your understanding here seems wrong because worker is actually a node or machine. Either you say it worker or worker node both are same

  1. My understanding - An executor is a sub-process(?) within a worker process. Each worker can have multiple executors. An executor is a process inside the worker node and a single worker node can have multiple executors

Q. What metric determines the number of executors per worker?

configuration(Number of cores and memory) of your worker node decides what is the max executors it can run on any specific worker node.

Q. Is the idea of a JVM associated with an executor process or at a higher "worker" level?

It is associated with the executor process. Spark executor is a single JVM instance on a node that serves a single spark application

Q. What is the relationship between a core and an executor?

Core property controls the number of concurrent tasks an executor can run. For example if you request 2 executor each with 2 cores then you can run 4 concurrent tasks at the same time during your job execution.

Q - Can RAM and HD be allocated at an executor level? For e.g., if I have a worker node with 100GB of RAM and 5 TB HD, can I allocate 20 GB RAM and 1 TB HDD per executor for that worker?

Generally spark perform all its computation in memory. RAM is allocated at the executor level and HD would be allocated at the Worker node level only. Spark would just spill the data to the disk only when it does not fit in memory

My understanding - A partition is a portion of the actual data. This split could happen using hashing, round robin or range. Q - What determines the location of these data partitions?

These partitions could be anywhere and might not be equally distributed in most of the cases.It could happen some of the executors does not have a single partition and other executors have more than 2 partitions.

In order to have colocated partitions or partitions that have same keys you would have to repartition data based on the specific column in your dataframe and then it would partition your data based on the values of that column and make sure that same column values are there in the same partition

When you repartition the data to 2 partitions then it would shuffle the data between all the executors and then break the dat into 2 partitions and then that data could be on any of the executors and other executors would be empty or idle in that case.

  1. Assumption - A task is the lowest unit of work that performs the actual ask. The number of tasks dependent on the number of partitions. So, if there are 20 partitions, I would have 20 tasks in each stage.

you would have 20 tasks for that specific stage and it wont remain same for all the stages as stage gets created when there is data shuffle that needs to happen. If there is no shuffle happening based on the code that you might have written it would just create a single stage with 20 tasks for sure.

Q - Are these tasks performed by individual executors? Yes

Q - If I have less executors (say, 10) than the partitions (say, 20), does it mean that only 10 tasks will be executed in parallel at any point? is the degree of parallelism constrained by the number of executors? Yes

user_48788480
Related