Spark Unit tests: Start spark session with HiveMetaStore running on an in-memory database

Viewed 733

We have written some Spark pipelines which uses DataFrames as well as some Hive SQLs (CREATE TABLE xxxx). Inside our unit tests, we need to start sparkSession with Hive support in local mode as follows:

SparkSession.builder()
         .master("local[*]")
         .config("spark.sql.catalogImplementation","hive")
         .enableHiveSupport()
         .getOrCreate()

I assume this spawn off a HiveMetaStore running locally on some port.I couldn't find much information on what happens internally in local mode. My questions are:

  1. Is my assumption correct about spawning a HiveMetaStore process?
  2. What kind of database does it use to store Hive metadata? Apache Derby?
  3. For the unit tests, is it possible to configure this to use some in-memory database like H2?
0 Answers
Related