(I've now solved this issue by adding a dependency as specified at the end of this post, but wondering if there is a better alternative or if I've missed something important?)
When trying to run a mapreduce job, the line
JobClient.runJob(conf)
gives the following error stack:
Exception in thread "main" java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:465)
at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:444)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:826)
My set-up is as follows:
public static void main(String[] args) throws IOException {
JobConf conf = new JobConf(Reduce.class);
conf.set("mapreduce.framework.name","yarn");
conf.set("mapreduce.jobhistory.address","s17.myserver.com:10020");
conf.set("mapreduce.jobhistory.webapp.address","s17.myserver.com:19888");
conf.set("yarn.resourcemanager.address","s6.myserver.com:8032");
conf.set("yarn.resourcemanager.scheduler.address","s6.myserver.com:8030");
conf.set("yarn.resourcemanager.resource-tracker.address","s6.myserver.com:8031");
conf.set("yarn.resourcemanager.admin.address","s6.myserver.com:8033");
conf.set("yarn.resourcemanager.webapp.address","s6.myserver.com:8088");
/// error on the following line
JobClient.runJob(conf);
}
After spending considerable time trying to check and re-check my configurations, I've managed to solve the problem by adding the following dependency to my project:
hadoop-mapreduce-client-jobclient
Am I missing something here or is the error message just a particularly misleading one?