How to refer pyspark.dbutils when developing code on local machine

user_19570330 0
Viewed 43

I'm new to Pyspark and asking question for best design pattern/practice:

I'm developing a library that should run both, on local machine and on Databricks.

Currently working on loading secrets. If code runs on databricks, I should load secrets using dbutils.secrets.get while if code runs on local machine, dotenv.load_dotenv.

Question: How can I create/refer to dbutils variable (which is readily provided in databricks instance)? pyspark doesnt have such module... even if I import SparkSession I still need DBUtils which is not found on pyspark local installation.

my current solution: if identify that code runs on Databricks, I create dbutils with:

dbutils = globals()['dbutils']
pyspark databricks python
1 Answers

Just follow the approach described in documentation for databricks-connect - wrap instantiation of dbutils into a function call that will behave differently depending on if you're on Databricks or not:

def get_secret(....):
  if spark.conf.get("spark.databricks.service.client.enabled") == "true":
    from pyspark.dbutils import DBUtils
    return DBUtils(spark).secrets.get(....)
  else:
    get local secret
Alex_Ott_186270
Related