My environment is Hadoop 3.2.1 and Spark 2.4.5.
Three executors, each executor has two cores and 1g memory.
Configuration about data locality is spark.locality.wait=3s.
And I have a question about data locality in the below picture.
I know Spark prefers to compute the nearest data. But slave1.lab.com compute time more than 32 seconds compared to the other two executors. Why it could happen, I already configured spark.locality.wait=3s.
And I reviewed all the tasks in this stage, they are all PROCESS_LOCAL.
Why other executors prefer to be idle rather than share computing tasks?