I have a change log table coming from the source database which is only producing New and Deleted records to capture changed data between two fetches. That means it will show New, deleted records in the latest fetch which were added, deleted after the previous fetch. To get updated records I have to use both New and Delete records, so I need to delete the record and add the new record. Do you think this change log table will have any issues when I need to model my slow changing dimensions and building other data models in snowflake?
This change log does not have any Updated records which have changed from the previous Fetch, but it is the combination of delete and a new record. Would merging this data cause any problems? I am able to model SCD 1 and SCD 2 using this change log, but was curious if any other issues would arise by not providing the updated records. For Eg:
Customer Master data:
cust_name, cust_id, change_status, Fetch_id
def, 2, NEW, 22
deg, 2, DEL, 22
deg, 2, NEW, 11
As you can see here the customer deg was updated in the new fetch "22", so to update this data they have to del and then insert with NEW