Spark SQL - hive is NOT enabled by default

Question

It seems hive is NOT enabled by default . It is started with "in-memory" How could i switch spark sql to make use of hive

This is on spark 2.4.5

scala> :paste
// Entering paste mode (ctrl-D to finish)

import org.apache.spark.sql.SparkSession
val spark = SparkSession
  .builder
  .enableHiveSupport()  // <-- enables Hive support
  .getOrCreate

// Exiting paste mode, now interpreting.

java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
  at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:869)
  ... 49 elided

scala>

scala> sql("set spark.sql.catalogImplementation").show(false)
+-------------------------------+---------+
|key                            |value    |
+-------------------------------+---------+
|spark.sql.catalogImplementation|in-memory|
+-------------------------------+---------+

Update - 1 no luck - with explicitly setting the value - spark.sql.catalogImplementation=hive

spark-shell --conf spark.sql.catalogImplementation=hive

Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id = local-1595914807895).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.5
      /_/

Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_232)
Type in expressions to have them evaluated.
Type :help for more information.

scala> sql("set spark.sql.catalogImplementation").show(false)
+-------------------------------+---------+
|key                            |value    |
+-------------------------------+---------+
|spark.sql.catalogImplementation|in-memory|
+-------------------------------+---------+

Update - 2 No luck , this time i specified jars

spark-shell   --jars     $HIVE_HOME/lib/hive-metastore-3.1.2.jar,    $HIVE_HOME/lib/hive-exec-3.1.2.jar,    $HIVE_HOME/lib/hive-common-3.1.2.jar,    $HIVE_HOME/lib/hive-serde-3.1.2.jar   --conf spark.sql.hive.metastore.version=3.1.2   --conf spark.sql.hive.metastore.jars=$HIVE_HOME"/lib/*"   --conf spark.sql.warehouse.dir=hdfs://localhost:9000/user/hive/warehouse --conf spark.sql.catalogImplementation=hive
    
    
    scala> sql("set spark.sql.catalogImplementation").show(false)
    +-------------------------------+---------+
    |key                            |value    |
    +-------------------------------+---------+
    |spark.sql.catalogImplementation|in-memory|
    +-------------------------------+---------+

scala> :paste // Entering paste mode (ctrl-D to finish)

import org.apache.spark.sql.SparkSession
val spark = SparkSession
  .builder
  .enableHiveSupport()  // <-- enables Hive support
  .getOrCreate


// Exiting paste mode, now interpreting.

java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
  at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:869)
  ... 57 elided

Spark SQL - hive is NOT enabled by default

0 Answers