Access Hive table from HDinsight cluster

user_7422128 0
Viewed 219

I am using pyspark to access hive inside my HDinsight cluster. When i go and query hive it shows me all the databases but when i query from spark it just shows default database.

I believe it just go and query spark catalog by default.

The workaround i found was i should use Hive warehouse connector to connect to hive from spark.

Is there any other way to do it?

Code

spark = SparkSession \
    .builder \
    .appName("Python Spark SQL Hive integration example") \
    .config("hive.metastore.uris", "thrift://hn0-mytestua.abc.dxbx.internal.cloudapp.net:9083") \
    .config("spark.sql.warehouse.dir", '/hive/warehouse/external') \
    .enableHiveSupport() \
    .getOrCreate()
spark.sql("show databases").show()
azure-hdinsight apache-spark hive hadoop python
1 Answers

If you don't want to specify hive related configuration in spark code then you can simply copy hive-site.xml file from $HIVE_HOME/conf foder to $SPARK_HOME/conf folder.

If it's not possible to do file copy then you can use below configurations while creating SparkSession or while launching spark jobs to connect to hive.

spark.sql.hive.metastore.jars = $HIVE_HOME/lib/* // No need to specify this if it's already in CLASSPATH
spark.hadoop.hive.metastore.uris = thrift://<host>:9083
spark.sql.hive.metastore.version= <hive version>

No need to specify metastore version if your hive version matches with default version specified here in documenation.

Azure HDInsight provides HWC connector to integrate hive with spark.

user_87733090
Related