I've been using SHA512 file hashes as a way to check for duplicate files prior to upload to a server. It's a very simple process of upload -> hash -> check hash against DB -> if not found, store. While this works well, the downside is that takes an average of 6 seconds to generate each hash due to file size.
I've been looking at using a checksum instead. During trial runs, I found SHA512 at around 6 seconds and murmur3f at around 0.5 seconds for the same file.
I would run the same piece of test code several times on various files and it was quite consistent. But I appear to have missed something... the test results depend heavily on which hash is run first.
If first is SHA512 it takes around 6s, with murmur3f following up with 0.5s.
If first is murmur3f it takes around 9s, with SHA512 following up with 6s. Subsequent re-runs on the same files puts murmur3f at a consistent 0.5s.
I have two questions:
- Is this the expected behaviour? (I'm assuming file read + caching.)
- If this behaviour is correct, if I'm not checking the same file over and over, why would I not just use SHA512?
This is all I'm doing to time it:
$time_start = microtime(true);
$checksum = hash_file( 'murmur3f', $file_path );
$time_end = microtime(true);