Azure Data Factory Dynamic content default values per environment

Question

I hope somebody can help me understand how I can solve this issue.

In azure I have 3 separate environment:

Development
Staging
Production

Each of this environment I have a Datafactory. Due to the large datasets I have I decided to build a CI CD pipeline to copy all the content of dev environment to staging and production every time I publish to adf_publish. This process work like a charm and I have all the datasets and linked services.

Here is where I start finding issues and I couldn't pass over them.

all 3 environments have cosmos databases with the exact same structure, but of course the url and access keys to cosmos are different.

Reading in Microsoft documentation, I have found that I can use dynamic content which is absolutely perfect. I created the parameters:

when I try the connection in dev environment, the parameters blade opens and I can set the default values I want and the connection succeded.

I did the same configuration in staging and after the release pipeline, I could see the Parameters,

But as you can see from the screenshot nothing else has been configured. My first thought was that maybe I need to configure the default values and apply the changes but even after that, after each release the values get reseted. Can anyone please help me understand how I can set the default values for each environment?

Please if there is any question or something I didn't explained well, just ask me and I will rephrase.

Thank you very much for any help you can provide me with.

UPDATE: Finally I managed to sort the first part of my problem, which is make the release pipeline set the dynamic content in the cosmos connection

But unfortunately my parameters value or they are hard coded or empty.

following this documentation https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment

I tried to create a arm-template-parameters-definition.json in the adf_publish in the root folder with this values:

{
"Microsoft.DataFactory/factories/linkedservices":{
    "properties": {
        "parameters": {
            "DBNameAggregated": {
                "type": "string",
                "defaultValue": "test"
            },
            "DBAccessKey": {
                "type": "string",
                "defaultValue": "test"
            }
        }
    }
}
}

when I saved and pushed the changes, I refreshed the portal and published. in my release pipeline I set the override param

-DBNameAggregated "test2"

but the release pipeline fails with the following error

Deployment template validation failed: 'The template parameters 'DBNameAggregated' in the parameters file are not valid; they are not present in the original template and can therefore not be provided at deployment time

I checked both my arm templates:

TemplateForFactory: Here I have the parameters declared.
TemplateParameters: Here the params are not declared

I am really hitting a wall here not knowing what I am doing wrong

Azure Data Factory Dynamic content default values per environment

0 Answers