How can I get a hash of an entire table in postgresql?

user_450128 0
Viewed 23538

I would like a fairly efficient way to condense an entire table to a hash value.

I have some tools that generate entire data tables, which can then be used to generate further tables, and so on. I'm trying to implement a simplistic build system to coordinate build runs and avoid repeating work. I want to be able to record hashes of the input tables so that I can later check whether they have changed. Building a table takes minutes or hours, so spending several seconds building hashes is acceptable.

A hack I have used is to just pipe the output of pg_dump to md5sum, but that requires transferring the entire table dump over the network to hash it on the local box. Ideally I'd like to produce the hash on the database server.

Finding the hash value of a row in postgresql gives me a way to calculate a hash for a row at a time, which could then be combined somehow.

Any tips would be greatly appreciated.

Edit to post what I ended up with: tinychen's answer didn't work for me directly, because I couldn't use 'plpgsql' apparently. When I implemented the function in SQL instead, it worked, but was very inefficient for large tables. So instead of concatenating all the row hashes and then hashing that, I switched to using a "rolling hash", where the previous hash is concatenated with the text representation of a row and then that is hashed to produce the next hash. This was much better; apparently running md5 on short strings millions of extra times is better than concatenating short strings millions of times.

create function zz_concat(text, text) returns text as 
    'select md5($1 || $2);' language 'sql';

create aggregate zz_hashagg(text) (
    sfunc = zz_concat,
    stype = text,
    initcond = '');
hash postgresql sql
7 Answers

Tomas Greif's solution is nice. But for huge enough table invalid memory alloc request size error will occur. So, it can be overcome with 2 options.

Option 1. Without batches

If the table is not big enough use string_agg and bytea data type.

select
    md5(string_agg(c.row_hash, '' order by c.row_hash)) table_hash
from
    foo f
    cross join lateral(select ('\x' || md5(f::text))::bytea row_hash) c
;

Option 2. With batches

If the query in previous option ends with error like

SQL Error [54000]: ERROR: out of memory Detail: Cannot enlarge string buffer containing 1073741808 bytes by 16 more bytes.

the row count limit is 1073741808 / 16 = 67108863 and the table should be divided to batches.

select
    md5(string_agg(t.batch_hash, '' order by t.batch_hash)) table_hash
from(
    select
        md5(string_agg(c.row_hash, '' order by c.row_hash)) batch_hash
    from
        foo f
        cross join lateral(select ('\x' || md5(f::text))::bytea row_hash) c
    group by substring(row_hash for 3)
    ) t
;

Where 3 in group by clause divides row hashes to 16 777 216 batches (2: 65 536, 1: 256). Also other batching methods (e.g. strictly ntile) will work.

P.S. If you need to compare two tables this post may help.

user_74728950
Related