I am using hadoop 2.10.x + hive 3.1.x + spark 3.0.1 and trying to load log file into hive by pyspark. I followed the code from spark document to connect to hive.
warehouse_location = abspath('spark-warehouse')
spark = SparkSession \
.builder \
.appName("Python Spark SQL Hive integration example") \
.enableHiveSupport() \
.getOrCreate()
but it always raise pyspark.sql.utils.IllegalArgumentException: <exception str() failed>.
Traceback (most recent call last):
File "log_extra.py", line 16, in <module>
.appName("Python Spark SQL Hive integration example") \
File "/usr/local/python37/lib/python3.7/site-packages/pyspark/sql/session.py", line 191, in getOrCreate
session._jsparkSession.sessionState().conf().setConfString(key, value)
File "/usr/local/python37/lib/python3.7/site-packages/py4j/java_gateway.py", line 1305, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/local/python37/lib/python3.7/site-packages/pyspark/sql/utils.py", line 134, in deco
raise_from(converted)
File "<string>", line 3, in raise_from
pyspark.sql.utils.IllegalArgumentException: <exception str() failed>
If I don't add config enableHiveSupport, this python script can run but can only connect to built-in hive.
I have put hive-site.xml in spark/conf.
Now I don't know how to connect to my hive by spark.