org.apache.hadoop.security.token.TokenIdentifier: Provider org.apache.hadoop.yarn.security.DockerCredentialTokenIdentifier not found

Question

While Connecting to kerberos hadoop cluster from the api deployed on tomcat to upload file to the hdfs, I am getting the above error. The keytab file is working fine on the cluster. I am able to Kinit and read the files in hdfs. Also the details in the krb5.conf is also as per the requirements. Default realm and all other required details are present in krb5.conf

Stack trace is,

Exception in thread "main" java.util.ServiceConfigurationError: org.apache.hadoop.security.token.TokenIdentifier: Provider org.apache.hadoop.yarn.security.DockerCredentialTokenIdentifier not found
    at java.util.ServiceLoader.fail(ServiceLoader.java:239)
    at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:372)
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
    at org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:117)
    at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:138)
    at org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$$anonfun$getTokenRenewalInterval$1$$anonfun$4.apply(HadoopFSDelegationTokenProvider.scala:116)
    at org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$$anonfun$getTokenRenewalInterval$1$$anonfun$4.apply(HadoopFSDelegationTokenProvider.scala:116)
    at scala.collection.TraversableLike$$anonfun$filterImpl$1.apply(TraversableLike.scala:248)
    at scala.collection.Iterator$class.foreach(Iterator.scala:891)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:247)
    at scala.collection.TraversableLike$class.filter(TraversableLike.scala:259)
    at scala.collection.AbstractTraversable.filter(Traversable.scala:104)
    at org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$$anonfun$getTokenRenewalInterval$1.apply(HadoopFSDelegationTokenProvider.scala:115)
    at org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$$anonfun$getTokenRenewalInterval$1.apply(HadoopFSDelegationTokenProvider.scala:111)
    at scala.Option.flatMap(Option.scala:171)
    at org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider.getTokenRenewalInterval(HadoopFSDelegationTokenProvider.scala:111)
    at org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider.obtainDelegationTokens(HadoopFSDelegationTokenProvider.scala:53)
    at org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:132)
    at org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:130)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.Iterator$class.foreach(Iterator.scala:891)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
    at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:206)
    at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
    at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
    at org.apache.spark.deploy.security.HadoopDelegationTokenManager.obtainDelegationTokens(HadoopDelegationTokenManager.scala:130)
    at org.apache.spark.deploy.yarn.security.YARNHadoopDelegationTokenManager.obtainDelegationTokens(YARNHadoopDelegationTokenManager.scala:59)
    at org.apache.spark.deploy.yarn.Client.setupSecurityToken(Client.scala:309)
    at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:1013)
    at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:178)
    at org.apache.spark.deploy.yarn.Client.run(Client.scala:1134)
    at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1526)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

user_19255208 · Answer

As the message in the exception clearly states,

Provider org.apache.hadoop.yarn.security.DockerCredentialTokenIdentifier not found

DockerCredentialTokenIdentifier class is missing on the classpath.

In my case when I started analyzing how does it know that the token should be resolved as a DockerCredentialTokenIdentifier and not something else, I found a file in the META_INF of the phoenix jar that had this class listed. We're using the hadoop classes from the hadoop jars, so deleted the META_INF file from the phoenix, which resolved the issue.

zip -d *.jar ./META-INF/services/org.apache.hadoop.security.token.TokenIdentifier

We were getting the same issue on the yarn when yarn is trying to launch the spark job, and the same steps helped resolving the issue in both places.

org.apache.hadoop.security.token.TokenIdentifier: Provider org.apache.hadoop.yarn.security.DockerCredentialTokenIdentifier not found

1 Answers