I'm writing a system for indexing the contents of files. The system monitors changes in files through the FileSystemWatcher and, by event, starts the process of updating the index.
Obviously, if file was changed, re-index whole file is bad solution, because the file can be large. It is necessary that the time spent for re-indexing depends on the number of changes and not on the file size.
The question is how quickly to determine what part of the file was changed? Calculating the diff between file versions is too resource-intensive.
I thought to do as follows:
- Split the file into parts by size of few kilobytes
- Calculate the digests (hashes) from each of the parts (SHA-1 or MD5)
- On the file change event quickly compare the digests of each part of the file
- Re-index only the changed parts of the file.
There are 2 problems here:
- The changed part (
#N) may contain shifts. These shifts affect the parts#N+1,#N+2, etc. This means that I will get other hashes for all these parts. If file was changed at the beginning, whole file will be re-indexed. - Hash functions give collisions, and therefore there is no guarantee that the coincidence of the digests means that the original data has not been changed.
Are there any ideas how this can be done at its best? Maybe there is a way to get the modified file system pages of the file?