I'm running a YARN + spark 2.4.5 cluster. I'm running the jobs using a docker as a yarn client mode. I configured the master and the docker client as follows:
yarn-site.xml
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>
<property>
<name>yarn.scheduler.fair.preemption</name>
<value>true</value>
</property>
<property>
<name>yarn.scheduler.fair.assignmultiple</name>
<value>true</value>
</property>
fairscheduler.xml:
<allocations>
<pool name="default">
<schedulingMode>FAIR</schedulingMode>
<maxRunningApps>2</maxRunningApps>
<fairSharePreemptionThreshold>0.6</fairSharePreemptionThreshold>
<fairSharePreemptionTimeout>1</fairSharePreemptionTimeout>
<weight>2</weight>
<minShare>2</minShare>
</pool>
<pool name="today">
<schedulingMode>FAIR</schedulingMode>
<maxRunningApps>2</maxRunningApps>
<fairSharePreemptionThreshold>0.6</fairSharePreemptionThreshold>
<fairSharePreemptionTimeout>1</fairSharePreemptionTimeout>
<weight>2</weight>
<minShare>2</minShare>
</pool>
<pool name="yesterday">
<schedulingMode>FAIR</schedulingMode>
<maxRunningApps>2</maxRunningApps>
<weight>1</weight>
<fairSharePreemptionThreshold>0.4</fairSharePreemptionThreshold>
<fairSharePreemptionTimeout>1</fairSharePreemptionTimeout>
<minShare>2</minShare>
</pool>
</allocations>
Now, I lunch two spark apps in two different containers. the first and then the second after a minute or two:
spark-submit --deploy-mode client --name job1 --queue today --py-files gs://babadod/spark.zip gs://babadod/spark/main.py --job job1 --job-args date=2020-04-02
spark-submit --deploy-mode client --name job2 --queue yesterday --py-files gs:/babadod/spark.zip gs://babadod/spark/main.py --job job2 --job-args date=2020-04-01
The output start on each of them:
20/04/02 13:52:08 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://namenode:4040
20/04/02 13:52:08 INFO FairSchedulableBuilder: Creating Fair Scheduler pools from /opt/spark/conf/fairscheduler.xml
20/04/02 13:52:09 INFO FairSchedulableBuilder: Created pool: default, schedulingMode: FAIR, minShare: 2, weight: 2
20/04/02 13:52:09 INFO FairSchedulableBuilder: Created pool: today, schedulingMode: FAIR, minShare: 2, weight: 2
20/04/02 13:52:09 INFO FairSchedulableBuilder: Created pool: yesterday, schedulingMode: FAIR, minShare: 2, weight: 1
20/04/02 13:52:09 INFO Utils: Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
20/04/02 13:52:09 INFO RMProxy: Connecting to ResourceManager at namenode/10.128.15.208:8032
20/04/02 13:52:09 INFO Client: Requesting a new application from cluster with 2 NodeManagers
The first app is taking all the available resource and the second starts after the first finishes. What do I miss? Thanks
EDIT
spark-defaults.conf:
spark.master yarn;
spark.dynamicAllocation.enabled true;
spark.shuffle.service.enabled true;
spark.yarn.shuffle.stopOnFailure false;