When running a map reduce job in a headless environment in MacOSX (e.g., when running jobs when ssh'ed as a particular user), I get the following exception or something like it…
2013-12-04 15:08:28,513 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1386194876944_0001 failed 2 times due to AM Container for appattempt_1386194876944_0001_000002 exited with exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
If instead, I log in as that user, the error does not occur and the MR job runs to conclusion, while a Java icon labeled "MRAppMaster" pops up in the dock.
I have narrowed this down to the ResourceManager starting a Java process without passing along the -Djava.awt.headless=true. When this occurs in a headless environment, the JVM does not have permission to display in the root window. This has showed up in a number of other circumstances, and I've corrected each.
This is not a question of permissions (something suggested elsewhere) or missing directories.
But I'm at a loss as to how to affect the last of the offending attempts to access the root window without permission.
I have added the -Djava.awt.headless=true option to the following:
- HADOOP_OPTS in hadoop-env.sh
- HADOOP_JOB_HISTORYSERVER_OPTS in mapred-env.sh YARN_OPTS in yarn-env.sh
- YARN_RESOURCEMANAGER_OPTS in yarn-env.sh (although that probably duplicates the YARN_OPTS
- mapred.{map|reduce}.child.java.opts and mapred.child.java.opts in mapred-site.xml
What am I missing? Might I be better off adding this to my Java options globally?
FYI, this is merely a pseudo cluster setup on a Mac, OS X 10.8.5, running Hadoop 2.2.0 downloaded from Apache with Java 1.6.0_65-b14. I did not use Homebrew or any other distribution. I am testing the pseudo-cluster with the WordCount example.
Thanks.
Ok. Mea culpa. I finally found all the settings to add… Search for all of the "opt" entries in the mapred-default.xml configuration instructions.
Here they are in-situ…
<property>
<name>mapred.child.java.opts</name>
<value>-Djava.awt.headless=true</value>
</property>
<!-- add headless to default -Xmx1024m -->
<property>
<name>yarn.app.mapreduce.am.command-opts</name>
<value>-Djava.awt.headless=true -Xmx1024m</value>
</property>
<property>
<name>yarn.app.mapreduce.am.admin-command-opts</name>
<value>-Djava.awt.headless=true</value>
</property>
I had also tried to accomplish the same thing by adding the parameter to _JAVA_OPTIONS in /etc/profile. Java picked it up, except when running the MRAppMaster!!!
Hopefully this helps someone else.