Databricks - Can we set environment variables for all clusters/using secret scopes in init scripts

user_18178534 0
Viewed 257

I would like to use a databricks secret in an init script. I can manually set an environment variable with ENVVAR={{/secrets/myscope/mysecret}} and then use ENVVAR in the init script. However doing it this way I would have to manually set that environment variable on each cluster, and any new ones, which won't work in my scenario.

Ideally I would like to manage it with cluster policies, but I don't see that being available there. I've also tried to have a separate init script with export ENVVAR="{{secrets/myscope/mysecret}}" But doing this when I try to echo that ENVVAR it returns nothing so I assume it is failing.

Does anyone have an idea on how I can achieve this?

Thanks Mat

databricks environment variables
1 Answers

You were on the right track by using an init script, but you cannot use normal export since then the variable will only be available to the init script subprocess.

Instead, use the following line in an init script to set an environment variable globally:

sudo echo AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY_HERE >> /etc/environment

This will write to the environment file of the cluster, which is read from any subprocess on the cluster.

Better way: Even better than setting the secret inside an init script is to use Databricks secrets API directly in code, e.g. like this:

dbutils.secrets.get(scope="myscope", key="mysecret")

Read more here: https://docs.databricks.com/dev-tools/databricks-utils.html#dbutils-secrets

user_46299500
Related