Databricks: how to get the data behind the workflow information page exported

user_6450450 0
Viewed 167

I would like to export all the handy data held on the databricks workflows tab shown herelovely workflow info

as a csv or anything really so I can use the data elsewhere. How do I go about that? NB I've tried just copy and pasting it as a last resort but it doesn't even do that well (one column of rubbish) so if can be done programmatically that would be great. Many thanks

azure-databricks databricks
1 Answers

There are few ways of achieving it:

  • One way is to use List jobs command of Jobs REST API as it mentioned by Ganesh - you can use whatever language you want to implement it, but you need to handle output correctly as there could be multiple pages of data.
  • Another way is to use jobs command of databricks-cli. The jobs list command has command-line parameter --all to get all defined jobs at once, and it also allows to get output as JSON, so you can use tool like jq to format it as you want (you must use --version 2.1 flag because this command requires API 2.1):
databricks jobs list --all --output JSON --version 2.1

P.S. If you need more detailed information, pass --expand-tasks parameter - it will output information about tasks inside jobs

Alex_Ott_186270
Related