I'm storing data in ADLS zones (Raw > Staging > Curated) which will be fed to a data warehouse. Think for example of Customer data from a CRM application:
Raw\CRM\Customer\2022\03\05\raw_crm_customer_2022_03_05.csv containing entries
1, John Smith,100
2, Mario Castillo,200
Raw\CRM\Customer\2022\03\06\raw_crm_customer_2022_03_06.csv containing entries:
2, Mario Castillo,300 //record has been modified
3, Mary Tyler, 500 // new record on 3rd March
So what should I store in the curated zone? What is the best naming convention?
Do I show an aggregation showing the latest modified version of each record? Or do I show historical records as well? For example, using aggregation on the last modified record:
1 John Smith, 100
2 Mario Castillo, 300
3 Mary Tyler,500
And what is the naming convention? Is it curated\CRM\Customer ?
My plan is to load historical data in the data warehouse from the curated zone.