Accessing S3 on AWS EMR when running application within Docker container via spark-submit

Viewed 845

I'm running an application that accesses data in an AWS S3 folder. The following script spark_emr_s3_test.py serves as a simple proxy for my application that reproduces my error:

from pyspark.sql import SparkSession
sess = SparkSession.builder.getOrCreate()
df = sess.read.csv('s3://path/to/my/dataset/folder')
print("Number of items: %d" % (df.count()))

Simple enough, and it doesn't fail when I run the following command...

spark-submit spark_emr_s3_test.py

... on an EMR cluster.

When I, however, try running the same application within a Docker container as described here in the documentation, I get the following error:

py4j.protocol.Py4JJavaError: An error occurred while calling o69.csv. : java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.amazon.ws.emr.hadoop.fs.EmrFileSystem not found

The exact command I use is the one in the above documentation link, but with values relevant to my case:

$ DOCKER_IMAGE_NAME=123456789123.dkr.ecr.us-east-1.amazonaws.com/emr-docker-examples:pyspark-example
$ DOCKER_CLIENT_CONFIG=hdfs:///user/hadoop/config.json
$ spark-submit --master yarn \
    --deploy-mode cluster \
    --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
    --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
    --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG \
    --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS=/etc/passwd:/etc/passwd:ro \
    --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
    --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
    --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG \
    --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS=/etc/passwd:/etc/passwd:ro \
    --num-executors 2 \
    spark_emr_s3_test.py

I can confirm that the error has nothing to do with any configuration or access management matter, as my script runs and the error pops up at the call to the CSV read function.

It appears to me that I can solve this by resolving the EMRFS dependencies within the container, or directly via spark-submit but I haven't been able to figure out how, and what exactly these dependencies are.

0 Answers
Related