How to combine sha hashes?

If you are trying to recreate the hash of a large payload (e.g. 1GB file) that has been split into chunks (e.g. 10MBs in size), the hash (MD5, SHA-256 etc.) needs to be computed on the entire collection. So using this example, you cannot add the 100 chunked hashes to recreate the hash of the original file. However...

You could send 2 values with each chunk:

• the individual chunk's hash (like you are doing now)
• the intermediate hash state, as your service sweeps through the file to create each chunk payload: at the beginning and end of the chunk

As the chunks are streamed in, one can verify the seams of the hash state at the end of chunk N matches that of the hash state at the beginning of chunk N+1.

The final hash state of the final chunk will be the hash for the entire payload.

Why do it like this? Because the hash can be computed in realtime as the file chunks are received - rather as a separate time-consuming process - after all the file chunks have been received.

Edit: based on comments:

Here's a crude state hash state solution:

Create a large random file (100MB):

dd if=/dev/urandom of=large.bin bs=1048576 count=100

Using an external tool to verify hash:

$ shasum -a 256 large.bin 
4cc76e41bbd82a05f97fc03c7eb3d1f5d98f4e7e24248d7944f8caaf8dc55c5c  large.bin

Running this playground code on the above file.

...
...
...
offset: 102760448   hash: 8ae7928735716a60ae0c4e923b8f0db8f33a5b89f6b697093ea97f003c85bb56  state: 736861032a24f8927fc4aa17527e1919aba8ea40c0407d5452c752a82a99c06149fd8d35000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006200000
offset: 103809024   hash: fbbfd2794cd944b276a04a89b49a5e2c8006ced9ff710cc044bed949fee5899f  state: 73686103bdde167db6a5b09ebc69a5abce51176e635add81e190aa64edceb280f82d6c08000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006300000
offset: 104857600   hash: 4cc76e41bbd82a05f97fc03c7eb3d1f5d98f4e7e24248d7944f8caaf8dc55c5c  state: 73686103c29dbc4aaaa7aa1ce65b9dfccbf0e3a18a89c95fd50c1e02ac1c73271cfdc3e0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006400000

the final hash matches.

Trying with an offset and intermediate hash-state. The file will be seeked to this offset, resuming the hash calculation from that point:

$ ./hash -o 102760448 -s "736861032a24f8927fc4aa17527e1919aba8ea40c0407d5452c752a82a99c06149fd8d35000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006200000"
offset:  103809024  hash: fbbfd2794cd944b276a04a89b49a5e2c8006ced9ff710cc044bed949fee5899f  state: 73686103bdde167db6a5b09ebc69a5abce51176e635add81e190aa64edceb280f82d6c08000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006300000
offset:  104857600  hash: 4cc76e41bbd82a05f97fc03c7eb3d1f5d98f4e7e24248d7944f8caaf8dc55c5c  state: 73686103c29dbc4aaaa7aa1ce65b9dfccbf0e3a18a89c95fd50c1e02ac1c73271cfdc3e0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006400000

we get the same final hash as before.

Note: this does expose the hash internal state, so be mindful of the security implications this may entail. With a large chunk-size, this should not be an issue.

edit: The purpose of this is to form a sort of a blockchain although that term is pretty overloaded these days. It's for integrity purposes, not proof of work. The idea is hash the blocks at the follower nodes, combine the hashes into one on the cluster leader to have a hash representing the chain as a whole, and then prepend that to the new blocks to be hashed.

this look like a merkle tree

https://brilliant.org/wiki/merkle-tree/

A Merkle tree is a hash-based data structure that is a generalization of the hash list. It is a tree structure in which each leaf node is a hash of a block of data, and each non-leaf node is a hash of its children. Typically, Merkle trees have a branching factor of 2, meaning that each node has up to 2 children.

Merkle trees are used in distributed systems for efficient data verification. They are efficient because they use hashes instead of full files. Hashes are ways of encoding files that are much smaller than the actual file itself. Currently, their main uses are in peer-to-peer networks such as Tor, Bitcoin, and Git.

Have a look to that implementation https://github.com/cbergoon/merkletree

Try taking your SHA256 hashes that you have already and put them in a string. Then hash that string using SHA256 or the Algorithm of your choice. Alternatively, you could string together your original chucks of data and then hash that. But I think its quicker and cleaner to just "hash the hashes".