Is there any relation between vCpu on Fargate resource and spark core/threads?

user_45280 0
Viewed 199

I'm running a spark batch job on aws fargate in standalone mode. On the compute environment, I have 8 vcpu and job definition has 1 vcpu and 2048 mb memory. In the spark application I can specify how many core I want to use and doing this using below code

sparkSess = SparkSession.builder.master("local[8]")\
    .appName("test app")\
    .config("spark.debug.maxToStringFields", "1000")\
    .config("spark.sql.sources.partitionOverwriteMode", "dynamic")\
    .getOrCreate()

local[8] is specifying 8 cores/threads (that’s what I'm assuming). Initially I was running the spark app without specifying cores and I think job was running in single thread and was taking around 10 min to complete but with this number it is reducing the time to process. I started with 2 it almost reduced to 5 minutes and then I have changed to 4, 8 and now it is taking almost 4 minutes. But I don't understand the relation between vcpu and spark threads. Whatever the number I specify for cores, sparkContext.defaultParallelism shows me that value.

Is this the correct way? Is there any relation between this number and the vcpu that I specify on job definition or compute environment.

aws-fargate pyspark apache-spark
1 Answers

You are running in Spark Local Mode. Learning Spark has this to say about Local mode:

Spark driver runs on a single JVM, like a laptop or single node

Spark executor runs on the same JVM as the driver

Cluster manager Runs on the same host

Damji, Jules S.,Wenig, Brooke,Das, Tathagata,Lee, Denny. Learning Spark (p. 30). O'Reilly Media. Kindle Edition.

local[N] launches with N threads. Given the above definition of Local Mode, those N threads must be shared by the Local Mode Driver, Executor and Cluster Manager.

As such, from the available vCPUs, allotting one vCPU for the Driver thread, one for the Cluster Manager, one for the OS and the remaining for Executor seems reasonable.

The optimal number of threads/vCPUs for the Executor will depend on the number of partitions your data has.

user_18593700
Related