I have a large (up to TB of data) datasets that needs to be transformed and sent/pulled from one database into another. It needs to work as initial full load and then incremental loads that would just contain data changed in the time period of concern.
The problem is how to ensure the database 2 has the same data as the database 1, except comparing it all which could be very time consuming. Is incremental sync reliable enough to run months and years with no data corruption?
What I was thinking about was to create hash of the data in DB 1 and DB 2 and compare them which should be faster than comparing it all.