Spark xxhash64 not consistent across execution environments

Question

I'm trying to use the Spark 3.0 DataFrame API built-in function xxhash64 to generate a long primary key column given a subset of columns like this:

df_with_id = df.withColumn(
        "CONTRACT_IDENTIFIER",
        xxhash64(
            "SOURCE_SYSTEM",
            "SOURCE_SYSTEM_CONTRACT_IDENTIFIER",
            "CONTRACT_INCEPTION_DATE",
        ),
    )

I have also created a unit test that compares the DataFrames with the generated IDs for equality. However, it looks like the generated hash code is not consistent across execution environments (JVMs?) because the test succeeds when run locally but fails when executed in an integration test Spark cluster.

So my question is whether this is something that is expected and whether there are other ways to achieve the same thing, i.e. create a row ID by hashing a subset of columns?

Spark xxhash64 not consistent across execution environments

0 Answers