CI / CD with Databricks Unity Catalog

Oscar Dyremyhr 0
Viewed 19

I am migrating tables from hive_metastore to Unity Catalog for my Databricks workspaces.

I have three databricks workspaces:

  • Dev
  • Test
  • Prod

Each workspace has its own ADLSv2 storage account. (Dev, test, prod)

Currently when developing I read in a table using

df = spark.table('bronze.my_table') # schema.table

This uses the default hive_metastore which points to the corresponding container (Workspace Dev -> Storage account Dev).

However, with Unity Catalog. It seems I would now have to specify the catalog too based on which workspace I work in. Unless, there is a default unity catalog for a workspace.

df = spark.table('dev.bronze.my_table') # catalog.schema.table

When deploying code from Dev -> Test -> Prod workspace. I would like to avoid having to dynamically set the catalog name for all notebooks using spark.table based on workspace (dev, test, prod). Basically 'bronze.my_table' when working in Dev points to delta table data stored in the dev catalog. While in Prod it points to delta table data stored in the prod catalog. Is this possible? I assume I can use the previous hive_metastore (one for each workspace) and build Unity Catalog on top of it (they reference each other and are in sync). However, isn't the idea that the Unity Catalog replaces the hive_metastore?

databricks-unity-catalog databricks
1 Answers

There are few approaches to this:

  1. At the beginning of your program issue the use catalog catalog_name SQL command, and then you can continue to use two-level naming for schema+table inside the catalog - continue to use df = spark.table('bronze.my_table')

  2. Incorporate the catalog name variable into table name, like, df = spark.table(f'{catalog_name}.bronze.my_table').

In all cases you need to either explicitly pass the catalog name as command-line option or widget or something like that, or try to map workspace URL to environment.

But really, it's recommended to pass table names as configuration parameters, so you can easily switch not only between catalogs, but also between schemas/databases.

Alex Ott0
Related