Cluster Matrics Details- Node = 1, vcores = 8,Memory = 25 GB
Scheduler Matrics- Min Allocation = memory 1024, vCores:1 ; Max Allocation = memory 3072, vCores:2
In spark-defaults.conf file spark.dynamicAllocation.enabled=true
I tried the command:
pyspark --master yarn --num-executors 2 --executor-cores 4 --executor-memory 3G
I am very new to Spark. So, I was just trying to understand how I can allocate memory and tasks. I tried it in cloudera cluster. But I am getting error as follows:
Required execution memory (3072), overhead (384), and PySpark memory (0 MB) is above the max threshold
(3072 MB) of the cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or
'yarn.nodemanager.memory-mb'.
When I tried with the below commands it is working:
pyspark --master yarn --num-executors 2 --executor-cores 2 --executor-memory 1G
pyspark --master yarn --num-executors 2 --executor-cores 2 --executor-memory 2G
I tried the following command:
pyspark --master yarn --num-executors 2 --executor-cores 2 --executor-memory 3G
it is also not working.
So, is it the reason that I can not cross the max allocation of Scheduler (memory 3072, vCores:2 in this case)? Can anyone please explain.