Spark on YARN + Secured hbase

I am submitting a job to YARN (on spark 2.1.1 + kafka 0.10.2.1) which connects to a secured hbase cluster. This job, performs just fine when i am running in "local" mode (spark.master=local[*]).

However, as soon as I submit the job with master as YARN (and deploy mode as client), I see the following error message -

Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user

I am following hortonworks recommendations for providing information to yarn cluster regarding the HBase and keytab etc. Followed this kb article - https://community.hortonworks.com/content/supportkb/48988/how-to-run-spark-job-to-interact-with-secured-hbas.html

Any pointers what could be going on ?

the mechanism for logging into HBase:

UserGroupInformation.setConfiguration(hbaseConf)
val keyTab = "keytab-location") 
val principal = "kerberos-principal"
val ugi = UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keyTab)
UserGroupInformation.setLoginUser(ugi)
ugi.doAs(new PrivilegedExceptionAction[Void]() {

override def run: Void = {
  hbaseCon = Some(ConnectionFactory.createConnection(hbaseConf))
  null
}
})

Also, I tried the alternative mechanism for logging in, as:

UserGroupInformation.loginUserFromKeytab(principal, keyTab)
connection=ConnectionFactory.createConnection(hbaseConf)

please suggest.