Azure data factory How to catch any error on any activity and log it into database?

user_14988166 0
Viewed 76

I am currently working on error handling in ADF. enter image description here

I know how to get error from a particular activity by using @ activity('').error.message and then pass it to database.

Do you guys know a generic way that able us to catch any error on any activity on a pipeline?

Best regards!

azure-data-factory error-handling
2 Answers

You can leverage the flow path dependency aspect within Azure data factory to manage logging of error based on single activity rather than duplicating same activities : enter image description here

The below blog : https://datasharkx.wordpress.com/2021/08/19/error-logging-and-the-art-of-avoiding-redundant-activities-in-azure-data-factory/ explains all the same in details

Below are the basic principles that need to be followed :

Multiple dependencies with the same source are OR’ed together. Multiple dependencies with different sources are AND’ed together. Together this looks like (Act_3fails OR Act_3Skipped) AND (Act_2Completes OR Act_2skipped)

user_114054230

When you have certain number of activities in pipeline1, and you want to capture any error message when one of the activities fail, then using execute pipeline activity is the correct approach.

  • When you use execute pipeline activity to trigger pipeline1, when any activity inside pipeline1 fails, it throws an error message which includes the name of the activity which failed.
  • Look at the following demonstration. I have a pipeline named p1 which has 3 activities get metadata, for each and Script.

enter image description here

  • I am triggering p1 pipeline in p2 pipeline using execute pipeline activity and storing the error message in a variable using set variable activity. For demonstration, I made sure each activity throws an error each time (3 times).
  • When the get metadata activity fails, the respective error message captured in p2 pipeline will be as follows:
Operation on target Get Metadata1 failed: The required Blob is missing. ContainerName: data2, path: data2/files/.

enter image description here


  • When only for each activity in p1 pipeline fails, the respective error message will be captured in p2 pipeline.
Operation on target ForEach1 failed: Activity failed because an inner activity failed

enter image description here


  • When only script activity in p1 pipeline fails, the respective error message will be captured in p2 pipeline.
Operation on target Script1 failed: Invalid object name 'mydemotable'

enter image description here

So, using Execute pipeline activity (to execute pipeline1) would help you to capture the error message from any different activities that fail inside the required pipeline.

user_188445850
Related