Collect Metadata with Dagster

user_13247155 0
Viewed 266

everyone

I've started using dagster for about a week or so now and I'm fascinated by the tool. However, I was wondering if it's possible to collect the metadata that is produced by dagster in the output.

The regular dagster output goes like this:

2021-06-17 15:12:30 - dagster - DEBUG - my_pipeline- 47989433-702c-4246-9c8d-ab4c8bab4be6 - 13936 - merge_transformations - LOADED_INPUT - Loaded input "clean_daag_df" using input manager "io_manager", from output "result" of step "clean_dzag"

[...]

2021-06-17 15:12:30 - dagster - DEBUG - my_pipeline - 47989433-702c-4246-9c8d-ab4c8bab4be6 - 13936 - merge_transformations - STEP_SUCCESS - Finished execution of step "merge_transformations" in 98ms.

I'd like to know how to access this information, specially the start and finish time of each solid as well as the pipeline run id and, if possible, the id of each solid execution. (instead of just seeing the output in the screen, I'd like to export it to a file or to a database).

Thanks in advance for any help.

dagster pipeline python
1 Answers

Dagster actually automatically stores these logs (in a structured format) for you. This is configurable by setting event_log_storage in your dagster.yaml file (so you can choose what type of database it uses), but by default they all get stored in a local Sqlite database in your $DAGSTER_HOME directory. Docs here: https://docs.dagster.io/deployment/dagster-instance#event-log-storage explain a bit more about how this works.

I'd also recommend checking out Dagit, which works with these stored event logs to help visualize past sold execution (among many other uses!).

user_161919200
Related