Hive query from pyspark

Viewed 85

I want to connect to hive from pyspark using following code:

from pyspark import SparkContext, SparkConf
from pyspark.conf import SparkConf
from pyspark.sql import SparkSession, HiveContext

sparkSession = (SparkSession
                .builder
                .master('spark://spark_host:7077')
                .appName('example-pyspark-read-and-write-from-hive')
                .config("spark.sql.warehouse.dir", "hdfs://spark_host:9000/user/hive/warehouse", conf=SparkConf())
                .enableHiveSupport()
                .getOrCreate()
               )

Output:

 raise QueryExecutionException(s.split(': ', 1)[1], stackTrace)
     78             if s.startswith('java.lang.IllegalArgumentException: '):
---> 79                 raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
     80             raise
     81     return deco

IllegalArgumentException: "Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder':"

My other attempts have failed. Please help me to configure pyspark and hive for normal working.

Spark version - 2.4.5 Hive version - 3.1.2

0 Answers
Related