Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog'

user_4751033 0
Viewed 695

I am unable to run Hive queries from Pyspark.

I tried copying hive-site.xml into spark's conf but inspite of doing that it is throwing the same error

Full rror

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/spark-2.4.0/python/pyspark/sql/context.py", line 358, in sql
    return self.sparkSession.sql(sqlQuery)
  File "/usr/local/spark-2.4.0/python/pyspark/sql/session.py", line 767, in sql
    return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
  File "/usr/local/spark-2.4.0/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
  File "/usr/local/spark-2.4.0/python/pyspark/sql/utils.py", line 79, in deco
    raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.IllegalArgumentException: u"Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog':"
pyspark hive
1 Answers

In my test with oozie, I had to add the Hive-related jars that Spark needs. Try adding the same in spark's conf maybe

user_108634660
Related