Azure Synapse: How to use Web Activity to Query REST API Pipeline Data?

user_6534818 0
Viewed 551

How can I use Synapse's Web Activity to query a pipeline run? In particular, I want to extract the error message in case of failure.

The initial setup is as follows, following the GET request posted by documentation: https://docs.microsoft.com/en-us/rest/api/synapse/data-plane/pipeline-run/get-pipeline-run#clouderror

https://i.stack.imgur.com/Ud14y.png

To get the RunId of the pipeline, I simply use this code: activity('Execute Pipeline1').output.pipelineRunId

When I inspect what was sent in the GET request, below, I see that it has indeed extracted a pipelineRunId, but not the one listed in the debug panel below.

https://i.stack.imgur.com/HaSHM.png

I suspect this is the issue, but how can I get the pipeline Run Id for the exact run that was ran and that is shown below?

Edit

Adding in pipeline Run IDs that I can query; but I cannot query the pipeline that was just ran.

https://i.stack.imgur.com/UcyO4.png

azure-synapse azure
2 Answers

Your pics show that your pipeline has not been published. Possible it is reading the pipelineId from the last published version? Also wouldn't it be better to use the System variable @{pipeline().RunId}?

user_26403380

I believe there is a confusion between the pipeline runID and activity runID.

enter image description here

Inorder to get the child pipeline runID you will have to use this dynamic expression - @activity('ExecutePipelineParent').output.pipelineRunId

As per the screenshots you have shared, seems like you are passing the correct child pipeline runID to the web activity.

I don't see any issue with extracting the pipeline runID. Incase if your web activity is failing, then you will have to go through the specific error message related to Web activity configuration to figure out the root cause.

user_114491800
Related