I'm trying to understand delta lake's structure of data flow from bronze, silver, gold. Gold is supposed to be for business usage and ready to ingest either by data warehouse or some reporting service. (Am I understanding that right?) My question is really for a more in-depth data lifecycle through ingestion into delta lake up to the export of these "gold" tables to data warehouse.
Thanks!
I'm still learning this myself but here's what I understand of it so far.
The bronze level is meant to be the data from source. It should be unchanged and simply saved to a delta table at the bronze level.
The silver level is first stage of cleaning. Here, you do your data governance, removal of nulls, etc.
The gold level is the final level of cleaned data that should be ready for use by different applications or ML platforms.
Between each step is a job that should be run on the previous level's data to clean it and prepare it for the next step.
Check out this link here for an in depth discussion: https://databricks.com/blog/2019/08/14/productionizing-machine-learning-with-delta-lake.html
