Azure Data Factory YAML deployment fails on overrideParameter key with space

user_2519368 0
Viewed 598

I have deployed data factory instances via yaml pipelines and multiline overrideParameters successfully in the past but it seems that the Azure DevOps parser does not like keys that have spaces in them and fails with error
There was an error while overriding [my-parameter] parameter because of 'TypeError: Cannot read property 'type' of undefined', make sure it follows JavaScript Object Notation (JSON).
The parameter name is automatically created via adf_publish branch so I cannot change it (I don't want to use custom templates if that is an option). Here is some sample setup and what I have tried.

adf_publish arm parameter file

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "factoryName": {
            "value": "my_adf"
        },
        "AAS-Shutdown-Trigger_properties_AAS Shutdown_parameters_AASServerName": {
            "value": "my_aas_server"
        }
    }
}

pipeline.yml

- task: AzureResourceManagerTemplateDeployment@3
  displayName: 'ARM Template deployment'
  inputs:
    deploymentScope: 'Resource Group'
    azureResourceManagerConnection: ${{ variables.azureSubscription }}
    subscriptionId: ${{ variables.subscriptionId }}
    action: 'Create Or Update Resource Group'
    resourceGroupName: ${{ variables.resourceGroupName }}
    location: 'West Europe'
    templateLocation: 'Linked artifact'
    csmFile: '$(System.DefaultWorkingDirectory)/adf_branch/${{ variables.dataFactoryRepoPath }}/ARMTemplateForFactory.json'
    csmParametersFile: '$(System.DefaultWorkingDirectory)/adf_branch/${{ variables.dataFactoryRepoPath }}/ARMTemplateParametersForFactory.json'
    overrideParameters: >-
      -factoryName "${{ variables.dataFactoryName }}"
      -AAS-Shutdown-Trigger_properties_AAS Shutdown_parameters_AASServerName "${{ variables.aasShutdownServer }}"
    deploymentMode: 'Incremental'

I have tried the following variants on the parameter key (nothing worked)

  • -AAS-Shutdown-Trigger_properties_AAS Shutdown_parameters_AASServerName
  • -"AAS-Shutdown-Trigger_properties_AAS Shutdown_parameters_AASServerName"
  • -'AAS-Shutdown-Trigger_properties_AAS Shutdown_parameters_AASServerName'
  • -["AAS-Shutdown-Trigger_properties_AAS Shutdown_parameters_AASServerName"]
  • -['AAS-Shutdown-Trigger_properties_AAS Shutdown_parameters_AASServerName']

If I remove the parameter in question, all works fine.

EDIT: Adding the adf resource definition of the trigger in question (as requested by @KamilNowinski

{
    "name": "AAS-Shutdown-Trigger",
    "properties": {
        "annotations": [],
        "runtimeState": "Started",
        "pipelines": [
            {
                "pipelineReference": {
                    "referenceName": "AAS Shutdown",
                    "type": "PipelineReference"
                },
                "parameters": {
                    "AASServerName": "my-aas",
                    "ResourceGroupName": "my-rg",
                    "EtlPipelineName": "ETL Pipeline",
                    "KeyVaultName": "my-kv",
                    "OMSIdSecretName": "OMS-Workspace-ID",
                    "OMSKeySecretName": "OMS-Workspace-Key",
                    "AutomationAccountName": "my-atm",
                    "RunbookName": "aas-runbook"
                }
            }
        ],
        "type": "ScheduleTrigger",
        "typeProperties": {
            "recurrence": {
                "frequency": "Minute",
                "interval": 10,
                "startTime": "2020-10-07T13:27:00Z",
                "timeZone": "UTC"
            }
        }
    }
}
azure-pipelines azure-data-factory azure-devops yaml
2 Answers

Looking at the AzureResourceManagerTemplateDeployment@3 code for PowershellParameters.ts, I don't think spaces in the parameter name can be handled.

The top if block of the parse method has the parser ending a symbol on a space (which is good, at least if you could handle quoting), but the isName method calls for there to be no "special" character in it (and quotes/brackets qualify as a special character by findLiteral).

I took the parser typescript and exercised it in a number of different ways, and none of them (including your variations) would allow that name to exist.

That doesn't mean there's no solution - what you could do is use an extension like Replace Tokens to change that parameter name in both the template and the parameters file to something simpler, then use overrideParameters to alter THAT name.

user_142896540

I'm posting an answer summarizing what we found so others have an easier time. As @WaitingForGuacamole has pointed out, it seems the ARM task source code cannot handle spaces in parameter keys irrespective of how one tries to quote it. So one would have to either introduce some function to replace spaces in parameter keys after ADF has published the ARM templates or one would have to pay attention not to have spaces in ADF pipeline names in the first place as pointed out by @KamilNowinski. I still believe it should not be that difficult for ADF to replace spaces (or any special character) with acceptable symbols before publishing. That still would leave the issue in case one would not use adf_publish ARM templates but rather edit the ADF json definitions directly.

user_25193680
Related