When I run LogisticRegression in Spark, I found a stage is special, as the number of executors increases, the average task processing time becomes longer, why it could happen?
Environment:
- All servers are local, no cloud.
- Server 1: 6 cores 10g memory (Spark master, HDFS master, HDFS slave).
- Server 2: 6 cores 10g memory (HDFS slave).
- Server 3: 6 cores 10g memory (Spark slave, HDFS slave).
- Deploy in Spark standalone mode.
- Input file size: Large enough, it can meet the requirements of parallelism. Spark will read the file from HDFS.
- All the workloads have the same input file.
You can see that only server3 will participate in the calculation(Only it will become Spark worker).
Special stage DAG
1 core 1g memory
spark-submit --executor-cores 1 --executor-memory 1g --total-executor-cores 1 ....
mid-task duration: 1s
2 core 2g memory
spark-submit --executor-cores 1 --executor-memory 1g --total-executor-cores 2 ....
mid-task duration: 2s
3 core 3g memory
spark-submit --executor-cores 1 --executor-memory 1g --total-executor-cores 3 ....
mid-task duration: 2s
4 core 4g memory
spark-submit --executor-cores 1 --executor-memory 1g --total-executor-cores 4 ....
mid-task duration: 3s
5 core 5g memory
spark-submit --executor-cores 1 --executor-memory 1g --total-executor-cores 5 ....
mid-task duration: 3s
As can be seen from the above figure, the more executors on a machine will cause the longer the average running time of a single task. May I ask why this could happen, and I had not seen the executor had disk spill, the memory should be sufficient.
Note: Only this stage will produce this phenomenon, other stages did not have this problem.