How to use secret keys in Azure Machine Learning Service pipelines

user_6361775 0
Viewed 2044

I am using Azure Machine Learning services and the pipeline functionality for data preparation, training and testing of my Machine Learning models. However, during my data preparation step, I need to connect to a database and I want to find a way to pass my secret passwords or keys without writing them in plain text in my script files.

Locally, I make use of environment variables for using secret passwords and keys, but to my best knowledge, this is impossible in the pipeline infrastructure, since Conda doesn't support passing environment variables. If anyone can confirm or deny this, it would be helpful.

In the Azure Machine Learning services in the Azure Portal, I have found a 'key vault' resource, that is created automatically when I create a 'Machine Learning service workspace' resource. This seems to be exactly what I need. Is it? And if so, how do I use it?

If neither of the above solves my issue, is there any other way to safely use secret passwords and keys in my scripts, without writing them in plain text in the scripts?

EDIT: I realize my question have a strong focus on database connections. However, the question is really about any kinds of secrets or passwords, not just database credentials. As have been pointed out in an answer, that could be worth mentioning here, is that Azure SQL database connections can (and should) be solved using the DataTransferStep.

azure-machine-learning-workbench azure-machine-learning-service
3 Answers

Instead of using environment variables, you could just pass the credentials via the argumentsparameter:

pipeline_step = PythonScriptStep(
    script_name='train.py',
    arguments=['--keyvault_name', 'MyKV', '--secret_name', 'MyPW'], ...

And define the script arguments in train.py as follows:

parser = argparse.ArgumentParser('train')
parser.add_argument('--keyvault_name')
parser.add_argument('--secret_name')
args = parser.parse_args()

You can then use the variables args.keyvault_name and args.secret_name in your script. You can use these values to read the password from the Key Vault. Of course, you must first create the Key Vault and store the password there. In addition, you must also ensure that AML Workspace has the permission to read secrets from the Key Vault.

Of course, you could also pass the password in the script argument in plain text, but this is not advisable.

user_64709150
user_57849830

Passing secrets to remote runs is now supported, as of Azure ML SDK version 1.0.57, through azureml.core.keyvault.KeyVault object:

See Using Secrets in Remote Runs section in this notebook

user_57849830
Related