Apacher Hadoop , Apacher Hive on Apache Spark JAVA_9 common-lang version

Viewed 1590

Trying set Hive use Spark as engine.

After long configurations got, Timeout error.

Application log file yarn logs -applicationId <app_id>

ERROR yarn.ApplicationMaster: User class threw exception: java.lang.NoSuchFieldError: JAVA_9
    java.lang.NoSuchFieldError: JAVA_9
        at org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:207)
        at org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)
        at org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:93)
        at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:370)

The Error is in file https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala

private val bufferCleaner: DirectBuffer => Unit =
if (SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_9)) {
  val cleanerMethod =
    Utils.classForName("sun.misc.Unsafe").getMethod("invokeCleaner", classOf[ByteBuffer])
  val unsafeField = classOf[Unsafe].getDeclaredField("theUnsafe")
  unsafeField.setAccessible(true)
  ..

After research found out that problem is in common-lang3* jar file. I updated all on files on server using with older versions to commons-lang3-3.9.jar jars in hadoop share , hive ,etc .

https://github.com/apache/commons-lang/blob/master/src/main/java/org/apache/commons/lang3/JavaVersion.java

But still getting this error .

0 Answers
Related