I want to connect to hive from pyspark using following code:
from pyspark import SparkContext, SparkConf
from pyspark.conf import SparkConf
from pyspark.sql import SparkSession, HiveContext
sparkSession = (SparkSession
.builder
.master('spark://spark_host:7077')
.appName('example-pyspark-read-and-write-from-hive')
.config("spark.sql.warehouse.dir", "hdfs://spark_host:9000/user/hive/warehouse", conf=SparkConf())
.enableHiveSupport()
.getOrCreate()
)
Output:
raise QueryExecutionException(s.split(': ', 1)[1], stackTrace)
78 if s.startswith('java.lang.IllegalArgumentException: '):
---> 79 raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
80 raise
81 return deco
IllegalArgumentException: "Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder':"
My other attempts have failed. Please help me to configure pyspark and hive for normal working.
Spark version - 2.4.5 Hive version - 3.1.2