Using the SQL Hive function md5() is generating a MD5 hash, with 32 characters. Is there a way in Hive to have the character limit reduced without compromising the integrity of the output? Unfortunately, I have a limit for certain columns that cannot be changed.
I am trying to mask some data fields and thought using a MD5 would be the best option, but am open to other methods. I know the rand() function can be used, but hoping to avoid it.
Use CRC32 or hash256 and truncate value to the desired length, this will be good enough if you do not need crypto-strong algorithm. sha256 truncated will have less probability of collisions, I'd recommend it.
Example:
select crc32('Some test message') AS CRC32,
sha2('Some test message', 256) as sha256,
substr(sha2('Some test message', 256),1,10) as sha256truncated
Result:
crc32 sha256 sha256truncated
111182007 43bf899ff002b5fa8e0510b22943ee5c15e467e26fe777d0740ba004ebf3a28d 43bf899ff0