Variable group yaml variable expression in Azure Devops Pipeline

Question

I'm having difficulties with variable expression with the AzureKeyVault task. Both the azureSubscription and KeyVaultName values expressions are not being resolved to have variables in a variable Group. It works if I define those variables manually in any way in the yaml. Either in azure-pipelines.yml or in the template variables_test.yml that I import at the beginning of the stage.

I read in the documentation, that variables from a variable group should be used like this $(var) but then it throws an error saying a subscription like '$(var)' does not exist.

Template: azure-pipelines.yml:

- stage: Test

  variables:
  - template: variables_Test.yml  # Template reference
  - group: testVariableGroup

  dependsOn: []

  jobs:

  - job: Test
    
    steps:

    - task: AzureKeyVault@1
      inputs:
        azureSubscription: ${{ variables.spn_kv_app }}
        KeyVaultName: ${{ variables.keyVaultName }}
        SecretsFilter: '*'

In the logs from the staging i can see the variables being set from the variable group:

Job preparation parameters
Variables:
  keyVaultName: $[ variablegroups.testVariableGroup.keyVaultName ]
  resourceGroupName: $[ variablegroups.testVariableGroup.resourceGroupName ]
  spn_cd_app: $[ variablegroups.testVariableGroup.spn_cd_app ]
  spn_kv_app: $[ variablegroups.testVariableGroup.spn_kv_app ]
  storageAccountName: $[ variablegroups.testVariableGroup.storageAccountName ]

Important thing to note that those variables are not set to secrets in the variable group.

PS. For anyone interested. I created tickets:

https://github.com/MicrosoftDocs/vsts-docs/issues/7586

https://developercommunity.visualstudio.com/content/problem/949342/azuresubscription-serviceconnection.html

If anything will be reslved i will post an answer.

user_7460777 · Answer

Variable group yaml variable expression in Azure Devops Pipeline

It should be work with $(var). Try to check following test yaml file:

Main Yaml:

trigger:
- master

variables:
- group: Test2

pool:
  vmImage: 'windows-latest'

stages:

- template: azure-pipelines.yml

The template azure-pipelines.yml:

stages:
- stage: Test

  dependsOn: []

  jobs:

  - job: Test

    steps:

    - task: AzureKeyVault@1
      inputs:
        azureSubscription: $(spn_kv_app)
        KeyVaultName: $(keyVaultName)
        SecretsFilter: '*'

And my Variables Group:

As the test result:

So, it works fine on my side. Please check your Variables Group name and the name of the Variables.

Hope this helps.

user_4953639 · Answer

I don't think this is a bug, but rather a product limitation. There are two kinds of fields you can't assign "external" (AKA from a var group) variables:

The ones that point to Service Connections, used in many different types of tasks, such as an ARM Template Deployment. (may have many different names depending on the task, such as "azureSubscription", "serviceConnection", etc)
The environment names, mandatory in in deployment jobs

This is actually documented, but in a very hidden place (should be on the main variables page!). See the last paragraph here.

You may say "But I used this in situation XYZ and it worked!" and I will agree with you, but that is very unpredictable, depends on the type of task. As a rule of thumb, I never use service connections and environments in variable groups.

But a workaround exists! Instead of using vargroups, you can use variables with static values in your main pipeline (which can later call as many templates as you like) or even use a "variable file" that will work just like an variable group, an external yaml template that contains those variables and its values. So, by doing this, you can keep these parameters in a single place, without having to look for them on every pipelines and templates in your solution.

Here's the most basic example:

main_pipeline.yml:

variables:
  - service_connection: my-service-connection-name
  - environment: Production

jobs:
  - deployment: JOB_DEPLOY
    environment: ${{ variables.environment }}
    strategy:
      runOnce:
        deploy:
          steps:
            - task: AzurePowerShell@5
              inputs:
                azureSubscription: '${{ variables.service_connection }}'
                ScriptType: 'FilePath'
                ScriptPath: '$(Build.Repository.LocalPath)/my_script.ps1'

You could also point to an external variable template:

main_pipeline.yml:

variables:
  - template: project-settings.yml@self

jobs:
  - deployment: JOB_DEPLOY
    environment: ${{ variables.environment }}
    strategy:
      runOnce:
        deploy:
          steps:
            - task: AzurePowerShell@5
              inputs:
                azureSubscription: '${{ variables.service_connection }}'
                ScriptType: 'FilePath'
                ScriptPath: '$(Build.Repository.LocalPath)/my_script.ps1'

project-settings.yml:

variables:
  - name: service_connection
    value: my-service-connection-name
  - name: environment
    value: Production

Both examples will work the same way. The important thing is to have the static values in a file, and use what is called "template expression" syntax (the ${{ }}). The reason for this is because template expressions get processed before running the pipeline, and at that time, only static values are available to the system. This makes those service connections and environments be able to get values for the variables at the time they need it.

Variable group yaml variable expression in Azure Devops Pipeline

PS. For anyone interested. I created tickets:

2 Answers