We are building a middleware {built using Node Js and postgresql} that will get data from Company A in a txt file, process/transform that data, and deliver it to Company B in form of a CSV file.
The issue is Company B only accepts incremental data (data that is new/changed) but Company A will send the full load.
We are thinking to create a hash-key for each record in a file and compare that whenever a new load is sent. The only issue is that there will be millions of records and will require a lot of space to store historical data.
Is there a smart way to tackle this type of use case, thanks? Maybe a python script or something.