We are having lot of | (pipe) separated flat files, which we process on daily basis in SQL Server using a SSIS package. Each flat file are divided into header section, content section and footer section. We regularly get newer version of the same files. We are trying to implement file comparison functionality between two versions of same file, to reduce the load of processing.
Which method will be more efficient ?
Storing both versions of same file into separate SQL Server tables with checksum column and filter out rows for which checksum values are not matching.
Implementing the similar checksum logic in C# or any other comparison algorithm available in C#.
You may suggest any other new algorithm to achieve the same.