Connect to Databricks managed Hive from outside

Question

I have:

An existing Databricks cluster
- Azure blob store (wasb) mounted to HDFS
- A Database with its LOCATION set to a path on wasb (via mount path)
- A Delta table (Which ultimately writes Delta-formatted parquet files to blob store path)
A kubernetes cluster
- Reads and writes data in parquet and/or Delta format within the same Azure blob store that Databricks uses (writing as delta format via spark-submit pyspark)

What I want to do:

Utilize the managed Hive metastore in Databricks to act as data catalog for all data within Azure blob store

To this end, I'd like to connect to the metastore from my outside pyspark job such that I can use consistent code to have a catalog that accurately represents my data.

In other words, if I were to prep my db from within Databricks:

dbutils.fs.mount(
  source = "wasbs://container@storage.blob.core.windows.net",
  mount_point = "/mnt/db",
  extra_configs = {..})

spark.sql('CREATE DATABASE db LOCATION "/mnt/db"')

Then from my Kubernetes pyspark cluster, I'd like to execute

df.write.mode('overwrite').format("delta").saveAsTable("db.table_name")

Which should write the data to wasbs://container@storage.blob.core.windows.net/db/table_name as well as register this table with Hive (and thus be able to query it with HiveQL)

How to I connect to the Databricks managed Hive from a pyspark session outside of Databricks environment?

user_3421652 · Answer

This doesn't answer my question (I don't think it's possible), but it mostly solves my problem: Writing a crawler to create tables from delta files.

Mount Blob container and create a DB as in question
Write a file in delta format from anywhere:

df.write.mode('overwrite').format("delta").save("/mnt/db/table")  # equivilantly, save to wasb:..../db/table

Create a Notebook, schedule it as a job to run regularly

import os
def find_delta_dirs(ls_path):
    for dir_path in dbutils.fs.ls(ls_path):
        if dir_path.isFile():
            pass
        elif dir_path.isDir() and ls_path != dir_path.path:
            if dir_path.path.endswith("_delta_log/"):
                yield os.path.dirname(os.path.dirname(dir_path.path))
            yield from find_delta_dirs(dir_path.path)


def fmt_name(full_blob_path, mount_path):
    relative_path = full_blob_path.split(mount_path)[-1].strip("/")
    return relative_path.replace("/", "_")


db_mount_path = f"/mnt/db"
for path in find_delta_dirs(db_mount_path):
    spark.sql(f"CREATE TABLE IF NOT EXISTS {db_name}.{fmt_name(path, db_mount_path)} USING DELTA LOCATION '{path}'")

Connect to Databricks managed Hive from outside

1 Answers