pyspark join on a secondary column when primary column finds no match

The following code is intended to do this operation, however, does not work. It still joins on both columns yielding two rows when I should only get one row.

df = df.join(
            msa,
            (msa.msa_join_key == df.metro_join_key)
            | (
                (msa.msa_join_key != df.metro_join_key)
                & (msa.msa_join_key == df.state_join_key)
            ),
            "left",
        )

This code should work, so I am not sure why it does not. I've seen similar questions that also cannot solve this given that they too yield additional rows. pyspark - join with OR condition

Is this a known bug in pyspark? Is there a different way to do this join?