Error "String is not a valid Base64-encoded string" while mounting Azure blob storage to Databricks using dbutils.fs.mount and dbutils.secrets.get

Terri S 0
Viewed 37

I'm following the documentation here to mount an Azure blob storage container to Databricks:

https://docs.databricks.com/_static/notebooks/data-sources/mount-azure-blob-storage.html

However, I get an error while attempting this.

Here is my code:

dbutils.fs.mount(
  source = "wasbs://log@my-storage.blob.core.windows.net",
  mount_point = "/mnt/log/",
  extra_configs = {"fs.azure.account.key.my-storage.blob.core.windows.net": dbutils.secrets.get(scope = "my-scope", key = "my-key")}
)

Here is the error:

shaded.databricks.org.apache.hadoop.fs.azure.AzureException: java.lang.IllegalArgumentException: The String is not a valid Base64-encoded string.

I am able to just run dbutils.secrets.get to retrieve the secret like so:

dbutils.secrets.get(scope = "my-scope", key = "my-key")

Output:

Out[12]: '[REDACTED]'

I expect this since it should not print the secret to the output. However, I would think I would get a permissions error or the like if I was not able to pass this value through to the extra_configs parameter.

Also, I am able to mount it easily if I directly insert the access key into the extra_configs parameter like so:

dbutils.fs.mount(
  source = "wasbs://log@my-storage.blob.core.windows.net",
  mount_point = "/mnt/log/",
  extra_configs = {"fs.azure.account.key.my-storage.blob.core.windows.net": "my-access-key")}
)

The problem is I can do this in DEV, but need to pass the secret instead in the upper environments.

Any help is much appreciated! Thanks!

mount databricks azure
1 Answers

Please follow below steps:

Step 1: Create Azure key vault and save Secrets name with access key:

enter image description here

Step 2: Create Secret scope and also please follow this syntax for creating secret scope in azure Databricks: https://<Azure_databricks_instance>#secrets/createScope and save DNS name and resource ID.

enter image description here

enter image description here

Create mount:

dbutils.fs.mount(
  source = "wasbs://<container_name>@<storage_account>.blob.core.windows.net",
  mount_point = "/mnt/<mount_data>",
  extra_configs = {"fs.azure.account.key.<Storage_account>.blob.core.windows.net":dbutils.secrets.get(scope = "<scope_name>", key = "scrate_key")})

Output:

enter image description here

BhanunagasaiVamsi0
Related