Dataproc: What is the primary use case of local Hive metastore?

user_2159158 0
Viewed 276

As a default, Dataproc uses the local MySQL (image versions 1.5+) database on the master node as the Hive table metadata store.

I do not fully understand the primary use case of this local metadata store.

What are the benefits of using it and the drawbacks of not using it?

google-cloud-dataproc
1 Answers

There are 3 deployment modes for Hive Metastore on Dataproc:

  1. In-cluster MySQL and Hive Metastore. This is the default. The lifecycle of Hive metadata (table schemas) is the same as the cluster. A typical use case is that you have input data in GCS and want the output data to be in GCS as well. In your Hive script, you first create external tables for the input and output data, then query the data from the input table with some transformations and insert the result into the output table. After the query is done, the table metadata is no longer needed.

  2. External MySQL, in-cluster Hive Metastore. In this deployment, you store Hive metadata in an external MySQL instance, typically Cloud SQL instance. The in-cluster Hive Metastore uses the external MySQL instance as the underlying database. See this doc for more details.

  3. External MySQL and Hive Metastore. This is the recommended mode. In this deployment, there are no MySQL and Hive Metastore in the cluster, Hive Server2 depends on an external Hive Metastore, typically Dataproc Metastore Service. See this doc for details.

Choose mode 1, when you don't need Hive metadata lifecycle to outlive the cluster lifecycle. Choose mode 2 or 3, when you need that.

user_4042640
Related