Unable to parametrize ML pipeline endpoint name - Azure Data Factory

user_5368122 0
Viewed 67

Sorry for long post, I need to explain it properly for people to undertsand.

I have a pipeline in datafctory that triggers a published AML endpoint: enter image description here

I am trying to parametrize this ADF pipeline so that I can deploy to test and prod, but on test and prod the aml endpoints are different.

Therefore, I have tried to edit the parameter configuration in ADF as shows here: enter image description here

Here in the section Microsoft.DataFactory/factories/pipelines I add "*":"=" so that all the pipeline parameters are parametrized:

 "Microsoft.DataFactory/factories/pipelines": {
        "*": "="
    }

After this I export the template to see which parameters are there in json, there are lot of them but I do not see any paramter that has aml endpoint name as value, but I see the endpint ID is parametrized.

enter image description here

My question is: Is it possible to parametrize the AML endpoint by name? So that, when deploying ADF to test I can just provide the AML endpoint name and it can pick the id automatically:

enter image description here

azure-machine-learning-service azure-data-factory-2 azure-data-factory azure
1 Answers

i faced the similar issue when deploying adf pipelines with ml between environments. Unfortunately, As of now, adf parameter file do not have ml pipeline name as parameter value. only turn around solution is modifiying the parameter file(json) file with aligns with your pipeline design. For example, i am triggering ml pipeline endpoint inside foreach activity-->if condition-->ml pipeline

Here is my parameter file values:

"Microsoft.DataFactory/factories/pipelines": {
    "properties": {
        "activities": [
            {
                "typeProperties": {
                    "mlPipelineEndpointId": "=",
                    "url": {
                        "value": "="
                    },
                    "ifFalseActivities": [
                        {
                            "typeProperties": {
                                "mlPipelineEndpointId": "="
                            }
                        }
                    ],
                    "ifTrueActivities": [
                        {
                            "typeProperties": {
                                "mlPipelineEndpointId": "="
                            }
                        }
                    ],
                    "activities": [
                        {
                            "typeProperties": {
                                "mlPipelineEndpointId": "=",
                                "ifFalseActivities": [
                                    {
                                        "typeProperties": {
                                            "mlPipelineEndpointId": "=",
                                            "url": "="
                                        }
                                    }
                                ],
                                "ifTrueActivities": [
                                    {
                                        "typeProperties": {
                                            "mlPipelineEndpointId": "=",
                                            "url": "="
                                        }
                                    }
                                ]
                            }
                        }
                    ]
                }
            }
        ]
    }
}

after you export the ARM template, the json file has records for your ml endpoints

"ADFPIPELINE_NAME_properties_1_typeProperties_1_typeProperties_0_typeProperties_mlPipelineEndpointId": {
        "value": "445xxxxx-xxxx-xxxxx-xxxxx"

it is lot of manual effort to maintain if design is frequently changing so far worked for me. Hope this answers your question.

user_199978780
Related