AWS EMR step Vs command line spark-submit

user_13491644 0
Viewed 810

What is the difference between submitting a EMR step as below vs running a spark submit on master node of the EMR cluster.

EMR step

aws emr add-steps --cluster-id j-2AXXXXXXGAPLF --steps Type=Spark,Name="Spark Program",ActionOnFailure=CONTINUE,Args=[--class,org.apache.spark.examples.SparkPi,/usr/lib/spark/lib/spark-examples.jar,10]

Spark Submit

spark-submit --master yarn --deploy-mode cluster my_spark_app.py my_hdfs_file.csv

Will running the spark submit directly on master node make it distributed between core nodes. What is the change in performance between these two methods?Which is a better approach

amazon-emr apache-spark
1 Answers

Submitting via EMR Step gives some additional monitoring and tooling on the AWS platform.

  • EMR has CloudWatch metrics for running/completed/failed steps.
  • EMR steps dispatch Eventbridge events on complete/failure which can be used as triggers.
  • If your EMR cluster is running on a private subnet, you'll have to tunnel to the subnet to monitor your jobs, EMR step status does not have this limitation.
  • Similar to above, if your cluster is on a private subnet, you'll have to tunnel in via ssh to call spark-submit, EMR API is publicly addressable.
  • EMR RunJobFlowStep has AWS Step Functions integration if you want to run an EMR job as part of a workflow via state machine.

I'm sure there are others but these are the benefits I've seen.

Edit: One caveat - with EMR steps you'll need to submit the job via command-runner.jar and these end up as running processes on your master node for the life of the EMR step. If you're running hundreds of steps, you may end up needing a larger master node to support all of these processes.

user_22805760
Related