I'm trying to deploy run a Python 3 Spark application on AWS ERM 5.5.0. I read a few articles on how to configure the cluster to require Python 3. I want to test that is getting set correctly so I created a simple application to print sys.version. Then I submit this job to the cluster.
When I run it on the master using either spark-submit --deploy-mode client /local/path/to/version.py or by submitting a step in client mode, it works and I see the version info (Python 3, yay!) on stdout.
When I run on the cluster (1 master, 1 core) using spark-submit --deploy-mode cluster /local/path/to/version.py or by submitting a step in cluster mode, it fails. The stderr log from the job says "Container exited with a non-zero exit code 1" in a few places. Looking this up, I was directed to yarn logs for the actual container logs. Looking at those, I see sets of logs for two containers.
In the stdout log for container container_1496243466754_0004_01_000001, I see that my actual python code ran and reported the python version (Python 3, yay again!). Since the code ran, I would expect the job to be a success, but it is not.
The logs from the other container container_1496243466754_0004_02_000001 show the following at the end of the stderr log:
17/05/31 17:11:58 INFO ApplicationMaster: Preparing Local resources
Exception in thread "main" java.io.FileNotFoundException: File does not exist: hdfs://ip-10-0-210-27.hwheel.in:8020/user/hadoop/.sparkStaging/application_1496243466754_0004/__spark_conf__.zip
at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$6.apply(ApplicationMaster.scala:160)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$6.apply(ApplicationMaster.scala:157)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.deploy.yarn.ApplicationMaster.<init>(ApplicationMaster.scala:157)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:765)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:67)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:66)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:66)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:764)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
I think that is causing my job to fail. Skipping back to stderr from the other container, I see:
17/05/31 17:11:52 INFO ApplicationMaster: Deleting staging directory hdfs://ip-10-0-210-27.hwheel.in:8020/user/hadoop/.sparkStaging/application_1496243466754_0004
So it appears to me that container_1496243466754_0004_01_000001 deleted the staging directory from HDFS when the application completes, but this trips up the other container that is then looking for a file in that directory 6 seconds later.
I've seen some (probably dated) explanations that this kind of error can occur due to Java version mismatches (apps in Java 8, spark in Java 7). However, I believe EMR runs all in Java 8 and that is the default on the cluster and I am not calling Java directly in my application.
How do I resolve this? Is my application just completing and deleting the staging files too quickly (it is just a single line of python)?
UPDATES:
- I added in a time.sleep(10) at the end of my one line program to make sure it was not a case of my program completing too quickly. No difference.
- I found documentation for spark.yarn.preserve.staging.files. I set this to true in spark-defaults.conf. I ran the same job again in cluster mode. This time both, containers run to completion w/ no errors. No problem w/ FileNotFound for spark_conf.zip. I can verify this file exists at the expected location on HDFS. However, the overall job still reports as failure
Application application_1496243466754_0007 failed 2 times due to AM Container for appattempt_1496243466754_0007_000002 exited with exitCode: 0. '0' seems like the correct exit code and the log files show no errors. Now I have no idea why the job is failing.