Unable to schedule oozie spark action with Kerberos cluster.
<workflow-app name="Spark Test" xmlns="uri:oozie:workflow:0.5">
<credentials>
<credential name="hiveCredentials" type="hcat">
<property>
<name>hcat.metastore.uri</name>
<value>${hcatMetastoreURI}</value>
</property>
<property>
<name>hcat.metastore.principal</name>
<value>${hcatPrincipal}</value>
</property>
</credential>
</credentials>
<start to="spark-4c1d"/>
<kill name="Kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="spark-4c1d" cred="hiveCredentials">
<spark xmlns="uri:oozie:spark-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<job-xml>/user/hive-site.xml</job-xml>
<configuration>
<property>
<name>oozie.action.sharelib.for.spark</name>
<value>spark</value>
</property>
</configuration>
<master>yarn</master>
<mode>cluster</mode>
<name>MySpark</name>
<class>com.test.testMain</class>
<jar>${nameNode}/user/test/test-20.0.0.jar</jar>
<spark-opts>
--conf spark.default.parallelism=16
--conf spark.driver.cores=1
--conf spark.driver.memory=1g
--conf spark.executor.cores=2
--conf spark.executor.instances=1
--conf spark.executor.memory=1g
--conf spark.sql.shuffle.partitions=16
--conf spark.sql.window.partitions=16
--conf spark.hadoop.hive.exec.dynamic.partition=true
--conf spark.hadoop.hive.exec.dynamic.partition.mode=nonstrict
--conf spark.sql.hive.convertMetastoreParquet=false
--conf spark.hadoop.fs.hdfs.impl.disable.cache=true
--conf spark.eventLog.enabled=false
--conf spark.yarn.maxAppAttempts=1
--conf spark.driver.extraJavaOptions="-Dlog4j.configuration=log4j-driver.properties -Dhdfs.url=${nameNode}"
--conf spark.executor.extraJavaOptions="-Dlog4j.configuration=log4j-executor.properties -Djava.util.logging.config.file=parquet.logging.properties"
--files /etc/hive/conf.cloudera.hive/hive-site.xml
--jars ${nameNode}/user/test/lib/*
</spark-opts>
</spark>
<ok to="End"/>
<error to="Kill"/>
</action>
<end name="End"/>
</workflow-app>
When I try to run the oozie action it fails with the following error.
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2070)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1982)
... 35 more
Failing Oozie Launcher, failure to login: for principal: hdfs javax.security.auth.login.LoginException: Unable to obtain password from user
The same workflow works fine if I use spark-submit inside a shell script with a custom keytab and principal as given in this link https://docs.cloudera.com/documentation/enterprise/5-8-x/topics/sg_spark_auth.html. Please suggest if I am missing anything