What is the fastest hash algorithm to check if two files are equal?

Background:

For file comparison, using a cryptographic-grade hash function such as MD5, SHA-1, SHA-2, SHA-3 etc will be very slow as these tools are optimised for good statistical and security properties over speed.

Unfortunately, many of the tools that some users will be familiar with use the cryptographic hash functions as this is probably the most widespread use of hashing from a users' perspective. So although you can use openssl dgst or sha1, sha256 etc to compare files, it will be very slow. This is especially the case for a large directory of large files, which also happens to be a very typical use case!

For internal applications, you may not be interested in cryptographic properties. Specifically, if you are worried that an adversary may be trying to create a collision on purpose, then you should stick with one of the above algorithms (and avoid MD5 or SHA-1 which are broken).

Benchmarking hash functions:

The SMhasher website has some benchmarks which aid direct performance comparison and notes / weakness, if you have specific needs.

Good compromise:

xxdhash is very fast (at the expense of security) and is perfect for file comparison tasks internally, when security is not of concern. Binaries are widely available and these include command line utilities.

Optimisation: You only need to run the hash on files of the same size: https://unix.stackexchange.com/questions/339491/find-a-file-by-hash

Example use case:

I want to check a large directory of photos to see if some duplicated files have crept in. I'm not integrating with the outside world and, in my use case, there is no chance that a malicious actor will try to add a non-duplicate photo with an identical hash (known as a collision).

Installation:

xxdhash is available in many distributions' repositories. To install on Debian-based distributions (including Ubuntu):

sudo apt update && sudo apt install xxhash

On OpenBSD:

doas pkg_add -U xxdhash

Or from github

Get unique hashes for a full directory of files:

The xxh128sum command line tool should now be available to you. You can combine this with the find command to look for duplicated files:

find . -type f -exec xxh128sum {} \; > hashes.txt

Find duplicates:

You now have a file of hashes and filenames that can be used to find duplicates. List just the filename of the second found duplicate:

awk 'visited[$1]++ { print $2 }' hashes.txt

You can do it all in a single step - why sort when none is needed until the duplicated list is found :

nice find . -type f -print0 \
\
| xargs -0 -P 8 xxh128sum --tag | pvZ -i 0.5 -l -cN in0 \
\
| mawk2 'BEGIN {

   _=(FS="(^XXH(32|64|128)[ ][(]|[)][ ]["(OFS="=")"][ ])")<"" 

  } __[$(NF=NF)]--<-_ ' |  pvZ -i 0.5 -l -cN out9 \
                                                  \
  | LC_ALL=C gsort -f -t= -k 3,3n -k 2,2 | gcat -n | lgp3 3

The same logic used to locate unique items in awk via its associative hashed array feature can also be leveraged to find duplicates - it's just the flip side of the same coin - only sort when the list is much smaller.

I got output like this in my own folder filled to the brim with duplicates :

 24131  =./songChunk_93634773_0011_.8045v202108091628512738.ts=56075211016703871f208f88839e2acc
 24132  =./songChunk_93634773_0011_.8045v202108091628512772.ts=56075211016703871f208f88839e2acc

 24133  =./songChunk_93634773_0011_.8045v202108091628512806.ts=56075211016703871f208f88839e2acc
 24134  =./songChunk_93634773_0011_.8045v202108091628512839.ts=56075211016703871f208f88839e2acc
 24135  =./songChunk_93666774_0043_.7102v202108091628512485.ts=77643645774287386a02e83808a632ed

 24136  =./songChunk_93666774_0043_.7102v202108091628536916.ts=77643645774287386a02e83808a632ed
 24137  =./songChunk_92647129_0023_.8045v202108091628536907.ts=146289716096910587a15001b5d2e9d6
 24138  =./songChunk_92647129_0023_.8045v202108091628536946.ts=146289716096910587a15001b5d2e9d6

That said, the major limitation of my lazy approach is that the first file with the same hash it sees is the one it keeps, so if you care about timestamps and naming and all that, then yes you'll have to do a side-by-side call to stat to get you all the precise timestamps and inode numbers and all that TMI it offers.

— The 4Chan Teller

I remember the old modem transfer protocols, like Zmodem, would do some sort of CRC compare for each block as it was sent. CRC32, if I remember ancient history well enough. I'm not suggesting you make your own transfer protocol, unless that's exactly what you're doing, but you could maybe have it spot check a block of the file periodically, or maybe doing hashes of each 8k block would be simple enough for the processors to handle. Haven't tried it, myself.