I am submitting a Spark job with spark-submit --master yarn --deploy-mode cluster to an EMR cluster on AWS, from a node outside the cluster.
The job has dependencies on the AWS SDK for S3.
I get ClassNotFoundException on org.apache.hadoop.fs.s3a.S3AFileSystem, unless I also set yarn.application.classpath in yarn-site.xml, on the client side (where I'm calling spark-submit). If I copy-paste this setting from the yarn-site.xml on the EMR cluster master node, it works.
What I don't understand is why I need to do this: if the cluster is already configured with this classpath, why do I also need to configure this property from the client side? Especially because all the paths refer to folders that live on the EMR cluster, not the client.