I'm new to Delta Lake and I would really appericiate if someone can help eveluate delta lake for a scenario. We have a customer data platform where we have two tables: customer, event.
We run queries like:
- Customers of India(customer table attribute filter) who Viewed This product(event "Product Viewed").
- Basically join of event and customer table.
Event table: is just a fact table. Once ingested, never updated.
But the customer table is being updated a lot:
- For example, customer attributes like "last page visited". Meaning a row might get updated many many times in minute while cusotmer is browsing our website.
- We update "Loyalty Score" attribute of customers every day morning.
Customer table might have roughly 1 billion plus records, 1 Million ingests/updates daily.
With such high rate of updates to same record in delta table, should we consider delta table or should we think of someting else.