Is there a way to safely store the internal state of an SHA256 hash

user_453851 0
Viewed 67

I'm looking for a way to store the internal state of an sha256 hash in python and then bring it back later so that I can continue hashing the stream of content where the code left off previously.

I suppose this might be possible via pickle but that raises a security risk because I want to store the state externally.

Is there any way to get at the internal bits/bytes of a hashlib.sha256() and then later create a new one from those same internal bytes. IE: this would work as if the later created one was made via the copy() method on the first.

A little more context

I'm uploading chunks of data to AWS S3 which will ultimately form a multipart upload. They're sent in a strict order and will initially be uploaded as S3 objects in their own right. When all have been uploaded they'll then be rolled into one object using upload_part_copy. At the end of the process I want to know the final SHA256 hash of the combined object.

I won't know the full file content until all parts are uploaded, so there's no way to know the SHA256 hash ahead of time. There's a requirement to make this process resumable. What I DON'T want to do is download 500GB of content just to re-compute the hashes.

My idea is that I could maintain a hashlib.sha256() object while I'm uploading then I might be able to extract the internal state of the hash object each time I upload a chunk of data and attach that as metadata to the chunk. If I need to resume then all I need to do is fetch the metadata on the last chunk and restore the hash object.

hashlib python
1 Answers

Rehash: Resumable Hashlib

https://github.com/kislyuk/rehash

Rehash is a resumable interface to the OpenSSL-based hashers (message digest objects) in the CPython hashlib standard library. Rehash provides hashers that can be pickled, persisted and reconstituted from their repr(), and otherwise serialized. The rest of the Rehash API is identical to hashlib.

Rehash hashers can be used to checkpoint and restore progress when hashing large byte streams:

import pickle, rehash
hasher = rehash.sha256(b"foo")
state = pickle.dumps(hasher)

hasher2 = pickle.loads(state)
hasher2.update(b"bar")

assert hasher2.hexdigest() == rehash.sha256(b"foobar").hexdigest()

__getstate__ and __setstate__

import rehash
hasher = rehash.sha256(b"foo")
state = hasher.__getstate__()

hasher2 = rehash.sha256()
hasher2.__setstate__(state)
hasher2.update(b"bar")
user_3099600
Related