What dependencies should be included when deploying a Spark application to EMR 6.x?

Viewed 433

I have a Spark application that I'm able to run locally. The dependencies I have are:

dependencies {
    implementation "org.scala-lang:scala-library:${scalaVersion}"

    implementation "org.apache.spark:spark-sql_2.12:${sparkVersion}"
    implementation "org.apache.spark:spark-launcher_2.12:${sparkVersion}"
    implementation "org.apache.spark:spark-catalyst_2.12:${sparkVersion}"
    implementation "org.apache.spark:spark-streaming_2.12:${sparkVersion}"
    implementation "org.apache.spark:spark-core_2.12:${sparkVersion}"
    implementation group: 'org.apache.spark', name: 'spark-mllib_2.12', version: "${sparkVersion}"
    implementation group: 'org.apache.spark', name: 'spark-hive_2.12', version: "${sparkVersion}"
    implementation group: 'org.apache.spark', name: 'spark-yarn_2.12', version: "${sparkVersion}"
    testImplementation group: 'org.apache.spark', name: 'spark-catalyst_2.12', version: "${sparkVersion}"


    implementation group: 'org.apache.hadoop', name: 'hadoop-aws', version: hadoop_version
    implementation group: 'org.mongodb.spark', name: 'mongo-spark-connector_2.12', version: '3.0.1'
    testImplementation "com.holdenkarau:spark-testing-base_2.12:${sparkVersion}_1.1.0"
}

When looking at the Spark docs, it lists a version matrix https://docs.amazonaws.cn/en_us/emr/latest/ReleaseGuide/emr-release-6x.html explaining what dependencies are provided.

The documentation also points to a separate Maven-compliant repository that should be used: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-artifact-repository.html and I'm unsure if that should be added like so (alongside Maven Central):

repositories {
      maven {
          url "https://s3.us-east-1.amazonaws.com/us-east-1-emr-artifacts/emr-6.2.0/repos/maven/"
      }
      mavenCentral()
}

From what I've understood, certain dependencies are already provided by EMR and those have custom Spark enhancements.

My cluster is created using the following command:

aws emr create-cluster --auto-scaling-role myprod-emr-auto-scaling --applications Name=Hadoop Name=Hive Name=Pig Name=Hue Name=Spark --bootstrap-actions '[{"Path":"s3://my-emr-bootstrap/sshkeys.sh","Name":"Add ssh keys"}]' --ebs-root-volume-size 20 --ec2-attributes '{"KeyName":"mridang_test","InstanceProfile":"myprod-emr","ServiceAccessSecurityGroup":"sg-xxxxxxxx","SubnetId":"subnet-xxxxxxxx","EmrManagedSlaveSecurityGroup":"sg-xxxxxxxx","EmrManagedMasterSecurityGroup":"sg-xxxxxxxx"}' --service-role myprod-emr-service --enable-debugging --release-label emr-6.2.0 --log-uri 's3n://aws-logs-0123456789-us-east-1/elasticmapreduce/' --name 'MridangTest' --instance-groups '[{"InstanceCount":1,"EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"SizeInGB":32,"VolumeType":"gp2"},"VolumesPerInstance":2}]},"InstanceGroupType":"CORE","InstanceType":"c5a.xlarge","Name":"Core - 2"},{"InstanceCount":1,"EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"SizeInGB":32,"VolumeType":"gp2"},"VolumesPerInstance":2}]},"InstanceGroupType":"MASTER","InstanceType":"c5a.xlarge","Name":"Master - 1"}]' --scale-down-behavior TERMINATE_AT_TASK_COMPLETION --region us-east-1

How should my build be configured so that I can correctly only include the necessary libs into my ZIP project? There's a fair amount of collateral around this topic but nothing cohesive that explains how to configure it all.

My setup is tied to Gradle unfortunately.

1 Answers

You can find those list of enhanced Jars in AWS document.

You don't need to create a fat-jar, because these jars are also added in the class path.

You can check the class paths via SSHing into the master instance. Location of class paths -

  • hive jars - /usr/lib/hive
  • hadoop jars - /usr/lib/hadoop
  • spark jars - /usr/lib/spark

You need to add the S3 location as maven repository in your gradle script, as you mention above. It's a public maven repository, so no credentials required.

Related