Azure Data Factory find which pipeline in a sequence failed

João Miguens 0
Viewed 60

I have a pipeline (PL_Main) containing a set of Execute pipeline activities in sequence, as the below image depicts: enter image description here Now, I was trying to create an activity (executed after PL_Main completion) that allowed me to understand if any of the 'childs' Execute Pipeline activities (PL_1, PL_2, PL3.1, PL_3.2) failed.

However, according my understanding thee pipelines output contains only the RunId and PipelineName. Is there a way that I can find what were the pipelines in my execution chain that failed (or at least the 1st one failling) and store their RunId in as a parameter for my next activity (New Activity) ?

Thanks in advance to everybody!

azure-data-factory-pipeline azure-data-factory azure-data-factory-2 azure
2 Answers

Assuming you need to pass the pipelinerunid of the failed activity within Main pipeline, you need to deign the pipeline in below way :

enter image description here

JSON:

{
"name": "pipeline8",
"properties": {
    "activities": [
        {
            "name": "Execute Pipeline1",
            "type": "ExecutePipeline",
            "dependsOn": [],
            "userProperties": [],
            "typeProperties": {
                "pipeline": {
                    "referenceName": "pipeline7",
                    "type": "PipelineReference"
                },
                "waitOnCompletion": true
            }
        },
        {
            "name": "Execute Pipeline2",
            "type": "ExecutePipeline",
            "dependsOn": [
                {
                    "activity": "Execute Pipeline1",
                    "dependencyConditions": [
                        "Succeeded"
                    ]
                }
            ],
            "userProperties": [],
            "typeProperties": {
                "pipeline": {
                    "referenceName": "pipeline9",
                    "type": "PipelineReference"
                },
                "waitOnCompletion": true
            }
        },
        {
            "name": "Execute Pipeline3",
            "type": "ExecutePipeline",
            "dependsOn": [
                {
                    "activity": "Execute Pipeline2",
                    "dependencyConditions": [
                        "Succeeded"
                    ]
                }
            ],
            "userProperties": [],
            "typeProperties": {
                "pipeline": {
                    "referenceName": "pipeline10",
                    "type": "PipelineReference"
                },
                "waitOnCompletion": true
            }
        },
        {
            "name": "Execute Pipeline4",
            "type": "ExecutePipeline",
            "dependsOn": [
                {
                    "activity": "Execute Pipeline2",
                    "dependencyConditions": [
                        "Succeeded"
                    ]
                }
            ],
            "userProperties": [],
            "typeProperties": {
                "pipeline": {
                    "referenceName": "pipeline11",
                    "type": "PipelineReference"
                },
                "waitOnCompletion": true
            }
        },
        {
            "name": "Fail1",
            "type": "Fail",
            "dependsOn": [
                {
                    "activity": "Execute Pipeline1",
                    "dependencyConditions": [
                        "Failed"
                    ]
                }
            ],
            "userProperties": [],
            "typeProperties": {
                "message": {
                    "value": "@activity('Execute Pipeline1').output.pipelineRunId",
                    "type": "Expression"
                },
                "errorCode": "1"
            }
        },
        {
            "name": "Fail2",
            "type": "Fail",
            "dependsOn": [
                {
                    "activity": "Execute Pipeline2",
                    "dependencyConditions": [
                        "Failed"
                    ]
                }
            ],
            "userProperties": [],
            "typeProperties": {
                "message": {
                    "value": "@activity('Execute Pipeline2').output.pipelineRunId",
                    "type": "Expression"
                },
                "errorCode": "2"
            }
        },
        {
            "name": "Fail3",
            "type": "Fail",
            "dependsOn": [
                {
                    "activity": "Execute Pipeline3",
                    "dependencyConditions": [
                        "Failed"
                    ]
                },
                {
                    "activity": "Fail5",
                    "dependencyConditions": [
                        "Skipped"
                    ]
                }
            ],
            "userProperties": [],
            "typeProperties": {
                "message": "@activity('Execute Pipeline3').output.pipelineRunId",
                "errorCode": "1"
            }
        },
        {
            "name": "Fail4",
            "type": "Fail",
            "dependsOn": [
                {
                    "activity": "Execute Pipeline4",
                    "dependencyConditions": [
                        "Failed"
                    ]
                },
                {
                    "activity": "Fail5",
                    "dependencyConditions": [
                        "Skipped"
                    ]
                }
            ],
            "userProperties": [],
            "typeProperties": {
                "message": "@activity('Execute Pipeline4').output.pipelineRunId",
                "errorCode": "4"
            }
        },
        {
            "name": "Fail5",
            "type": "Fail",
            "dependsOn": [
                {
                    "activity": "Execute Pipeline3",
                    "dependencyConditions": [
                        "Failed"
                    ]
                },
                {
                    "activity": "Execute Pipeline4",
                    "dependencyConditions": [
                        "Failed"
                    ]
                }
            ],
            "userProperties": [],
            "typeProperties": {
                "message": {
                    "value": "@concat(activity('Execute Pipeline3').output.pipelineRunId,';',activity('Execute Pipeline4').output.pipelineRunId)",
                    "type": "Expression"
                },
                "errorCode": "5"
            }
        }
    ],
    "variables": {
        "test": {
            "type": "String"
        },
        "spt": {
            "type": "Array"
        },
        "final": {
            "type": "String"
        }
    },
    "annotations": []
}

}

Reference blog : https://datasharkx.wordpress.com/2021/08/19/error-logging-and-the-art-of-avoiding-redundant-activities-in-azure-data-factory/

Note: the fail activities capture the pipelinerunid

Finaly to capture the ID: enter image description here

@activity('Execute Pipeline1').error.message
Nandan0

You can try this approach also, to get the failed pipeline run id.

For this name the Execute pipeline activities name same as child pipeline names.

First get the Failed Execute pipeline activity name (Pipeline name) in the main pipeline using the error message.

Error message:

enter image description here

Use set variable to get this.

enter image description here

Get the pipeline name from the error message.

use the below expression for that.

@substring(variables('error'), add(indexOf(variables('error'),'target'),7), sub(indexOf(variables('error'),' failed'),add(indexOf(variables('error'),'target'),7)))

enter image description here

You can see that we got the pipeline name in the variable

enter image description here

Now get all the pipeline runs from the Pipeline Runs - Query By Factory command.

POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/queryPipelineRuns?api-version=2018-06-01

Use this in web activity.

This will give you all pipeline runs details with names and run ids Json which you can see in the above documentation.

Now compare our pipeline name from variable in this JSON and get that particular run id which is our required failed pipeline run id. Use ForEach activity or filter for that.

Reference:

Check this SO thread for reference by KarthikBhyresh-MT

Rakesh Govindula0
Related