Add custom parameters to Azure Data Factory deployment

user_12960708 0
Viewed 502

I need a help to access a linked service parameter during Azure Data Factory deployment or find some other way to set a parameter during deployment even if the parameter is not automatically added for editing.

I am using continuous integration for Azure Data Factory using Azure DevOps pipeline (i.e. all pipelines and connections are first created in a test resource and then deployed through Azure DevOps pipeline to production resource, https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery). For authentication I use Key Vault but Databricks Workspace URL cannot be added as a secret from Key Vault. I created a parameter for the value (DatabricksUrl) but I am not able to access that parameter during deployment because it is only created on the linked service. Only parameters added to ARMTemplateParametersForFactory.json file in the publish branch can be accessed. Is there a way to solve this? Any help appreciated.

ADB_linked_service_in_ADF

azure-data-factory-2 databricks azure-devops azure deployment
2 Answers

To be able to edit values that are not automatically available during the deployment, you have to add them to the ARM template parameters.

  1. go to the Manage tab in Azure Data Factory,
  2. select ARM template,
  3. click on edit parameter configuration,
  4. in the part "Microsoft.DataFactory/factories/linkedServices" there is specification for parameter extraction for all linkd services (marked with *) - below you can add the following which will allow you to edit the Databricks domain during deployment:
"AzureDatabricks": {
            "properties": {
                "typeProperties": {
                    "domain": "="
                }
            }
        }
  1. you need to save it (by clicking OK) and publish the Data Factory (you will not see it among changes for publish but it will affect the ARM template generation)

Then you will be able to override the value during Azure DevOps pipeline deployment.

More information on ARM template parameters https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-resource-manager-custom-parameters

user_129607080

It is easy to be lost with parameters and variables in Azure Data Factory. Good idea can be to specify it via global parameters and use it as central control place for variables:

enter image description here

user_109729590
Related