Register Azure ML Model from DatabricksStep

Viewed 620

I'm calculating a model while executing a DatabricksStep in an Azure ML Pipeline, save it on my Blob Storage as .pkl file and upload it to the current Azure ML Run using Run.upload_file (). All this works without any problems.

But as soon as I try to register the model to the Azure ML Workspace using Run.register_model (), the script throws the following error:

UserErrorException: UserErrorException: Message: Operation returned an invalid status code 'Forbidden'. The possible reason could be:

  1. You are not authorized to access this resource, or directory listing denied.
  2. you may not login your azure service, or use other subscription, you can check your default account by running azure cli commend: 'az account list -o table'.
  3. You have multiple objects/login session opened, please close all session and try again.

InnerException None ErrorResponse { "error": { "code": "UserError", "message": "\nOperation returned an invalid status code 'Forbidden'. The possible reason could be:\n1. You are not authorized to access this resource, or directory listing denied.\n2. you may not login your azure service, or use other subscription, you can check your\ndefault account by running azure cli commend:\n'az account list -o table'.\n3. You have multiple objects/login session opened, please close all session and try again.\n " } }

with the following call stack

/databricks/python/lib/python3.7/site-packages/azureml/_restclient/models_client.py in register_model(self, name, tags, properties, description, url, mime_type, framework, framework_version, unpack, experiment_name, run_id, datasets, sample_input_data, sample_output_data, resource_requirements) 70 return self.
71 _execute_with_workspace_arguments(self._client.ml_models.register, model, ---> 72 custom_headers=ModelsClient.get_modelmanagement_custom_headers()) 73 74 @error_with_model_id_handling

/databricks/python/lib/python3.7/site-packages/azureml/_restclient/workspace_client.py in _execute_with_workspace_arguments(self, func, *args, **kwargs) 65 66 def _execute_with_workspace_arguments(self, func, *args, **kwargs): ---> 67 return self._execute_with_arguments(func, copy.deepcopy(self._workspace_arguments), *args, **kwargs) 68 69 def get_or_create_experiment(self, experiment_name, is_async=False):

/databricks/python/lib/python3.7/site-packages/azureml/_restclient/clientbase.py in _execute_with_arguments(self, func, args_list, *args, **kwargs) 536 return self._call_paginated_api(func, *args_list, **kwargs) 537 else: --> 538 return self._call_api(func, *args_list, **kwargs) 539 except ErrorResponseException as e: 540 raise ServiceException(e)

/databricks/python/lib/python3.7/site-packages/azureml/_restclient/clientbase.py in _call_api(self, func, *args, **kwargs) 234 return AsyncTask(future, _ident=ident, _parent_logger=self._logger) 235 else: --> 236 return self._execute_with_base_arguments(func, *args, **kwargs) 237 238 def _call_paginated_api(self, func, *args, **kwargs):

/databricks/python/lib/python3.7/site-packages/azureml/_restclient/clientbase.py in _execute_with_base_arguments(self, func, *args, **kwargs) 323 total_retry = 0 if self.retries < 0 else self.retries 324 return ClientBase._execute_func_internal( --> 325 back_off, total_retry, self._logger, func, _noop_reset, *args, **kwargs) 326 327 @classmethod

/databricks/python/lib/python3.7/site-packages/azureml/_restclient/clientbase.py in _execute_func_internal(cls, back_off, total_retry, logger, func, reset_func, *args, **kwargs) 343 return func(*args, **kwargs) 344 except Exception as error: --> 345 left_retry = cls._handle_retry(back_off, left_retry, total_retry, error, logger, func) 346 347 reset_func(*args, **kwargs) # reset_func is expected to undo any side effects from a failed func call.

/databricks/python/lib/python3.7/site-packages/azureml/_restclient/clientbase.py in _handle_retry(cls, back_off, left_retry, total_retry, error, logger, func) 384 3. You have multiple objects/login session opened, please close all session and try again. 385 """ --> 386 raise_from(UserErrorException(error_msg), error) 387 388 elif error.response.status_code == 429:

/databricks/python/lib/python3.7/site-packages/six.py in raise_from(value, from_value)

Did anybody experience the same error and knows what is its cause and how to solve it?

Best, Jonas

UPDATE:

 model = sklearn.linear_model.LinearRegression ( )
 model_path = "<path to 'model.pkl' in my blob storage>"
 joblib.dump(model, model_path)
 aml_run = azureml.core.get_context ( )
 aml_run.upload_file (name = "model.pkl", path_or_stream = model_path)
 # Until this point, everything works fine
    
 aml_run.register_model (model_name = "model.pkl")
 # This throws the posted "Forbidden"-Error
2 Answers

To configure the workspace to authenticate to the subscription, Please follow the steps in the notebooks..

  • Persist model (joblib.dump) to a custom folder other than outputs.
  • Manually run upload_file to upload the model AML workspace. Name the destination same name with your model file.
  • Then run run.register_model.

or AML background process to automatically upload content under ./outputs to AML workspace. Once the upload is complete and call run.register_model which takes the content from AML workspace.

The documentation DatabricksStep class and the sample notebook https://aka.ms/pl-databricks should both be helpful.

UserErrorException: UserErrorException: Message: Operation returned an invalid status code 'Forbidden'.

This error might be due to the fact that Azure Databricks Compute is unable to authenticate the Azure Machine Learning Workspace.

I had been facing a similar error, and this is the Microsoft preferred way of solving this issue:

  1. Create an Azure Key Vault.
  2. Create a Service Principal (App registration) inside Azure Active Directory.
  3. Add this Service Principal with Contributor/ Owner access in AML and ADB.
  4. Create an Azure Databricks Scope and link it with the key vault created in Step 1.
  5. Save the Client ID, Directory ID and Client Secret in the Key Vault.
  6. Use ServicePrincipalAuthentication to validate the credentials.

For Step Six use Databricks Secret Scope to get the values. This resource will walk you through this step: Secret Management in Azure Databricks

Some references that will be helpful:

  1. A worked-out example provided by Microsoft.
  2. Microsoft Documentation on ServicePrincipalAuthentication
Related