Monitoring data factory pipeline runs using az cli not working as expected, and missing pipeline runs?

user_19219921 0
Viewed 201

I am trying to get the latest pipeline run logs from a azure data factory, where pipelines run every 10 minutes or so.

I am using the following az-cli command :

 az datafactory pipeline-run query-by-factory --factory-name XXXXX --last-updated-after $last_updated_after --last-updated-before $last-updated-before --resource-group XXXXX

Where $last_updated_after and $last_updated_before are defined like the following :

(Python code sample)
last_updated_after = (datetime.datetime.now() - datetime.timedelta(hours=3)).isoformat()
last_updated_before = (datetime.datetime.now()).isoformat()

The documentation isn't very clear what these two parameters represent, but they're supposed to work like a time window filter for selecting our pipeline runs.

However, I only get pipeline runs that occured X hours ago, as defined by last_updated_after, and won't get the pipeline runs that occured less than X hours ago.

And If I use hours =2 or less, I won't get any results, even though there are pipelines running in the past two hours.

Is this a bug? Is there a solution or an alternative?

azure-cli azure-data-factory azure
1 Answers
  • We have also tried Monitoring data factory pipeline runs using az cli commands in our environment.
  • We also got null values when we given last-updated-after , last-updated-before respective values, but after using filter on pipeline then got the results.
  • There is some lag in time(iso) format when the pipeline actually ran and the time what we are getting through az cli command.

Yes, we do have an alternate solution to solve this issue to monitor all your pipeline runs directly.

  • Select the Monitor & Manage tile on the Azure portal's data factory blade to launch the monitoring experience.
  • If you're already in the ADF UX, go to the left sidebar and click the Monitor icon.
  • This is the easiest way to monitor all your pipeline runs directly.

enter image description here

Reference: Visually monitor Azure Data Factory

To raise a support ticket for the support engineer to investigate more on this.

user_180436990
Related