I want to create a report with live status of Power BI dataset refreshes + history of the refreshes. Power BI has a REST API, with Python I can get the data, but now I'm stuck. Since I want to intervene immediately if a dataset refresh fails, the data must be (near) real-time.
Could I do this with databricks? I check whether there is new data in a blob storage and then I load that data into the dwh.
Or is there perhaps a better approach for this project?