NoClassDefFoundError: org/apache/spark/AccumulatorParam with spark execution engine on hive

Viewed 472

Trying to use SPARK as hive execution engine, Getting below error

Job failed with java.lang.ClassNotFoundException: org.apache.spark.AccumulatorParam
FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. java.lang.NoClassDefFoundError: org/apache/spark/AccumulatorParam
    at org.apache.hive.spark.counter.SparkCounterGroup.createCounter(SparkCounterGroup.java:52)
    at org.apache.hive.spark.counter.SparkCounters.createCounter(SparkCounters.java:71)
    at org.apache.hive.spark.counter.SparkCounters.createCounter(SparkCounters.java:67)
    at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:337)
    at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:358)
    at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:323)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.AccumulatorParam
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 10 more

  • Spark versions : 3.1.1
  • Hive version : 3.1.2 (also tried with - version2.3.8 got same error though)
  • Hadoop version : 3.3.0
  • 0 Answers
    Related