Building a function that can derive a new column of hashes based on other pd.DataFrame features

user_2780360 0
Viewed 246

I have a pandas DataFrame of health insurance information - names, addresses, DOBs, etc.

I wrote a function that works on a single row:

    
def make_hash(partner: str, df: pd.DataFrame) -> str:
  """
  For Partner A, df (pd.DataFrame) must contain:
    health_plan_id: str 
    date_of_birth: dt.Timestamp
    first_name: str
  Other partners will have different feature names for hash input and require a new elif block, below.
  """
  if partner == 'Partner A':
    health_plan_id = str(df.loc[:,'ID'].item()).strip().encode()
    date_of_birth = str(dt.date(df.loc[:,'Date of Birth'].item())).encode()
    first_name = str(df.loc[:,'Member Name'].item()).split(",")[1].strip().encode()

    hash_input = health_plan_id + date_of_birth + first_name
    h = hashlib.sha256(string=hash_input).hexdigest()
    print(f"Input: {hash_input}. Result: {h}.\n")
    return h
  else:
    print("No hashing strategy defined for that partner.")

Which outputs (values changed for PII):

make_hash(partner="Partner A", df=df)

Input: b'B88845204081984-06-11MickeyMouse'. Result: 4d578e1acd7c670193448b84362095383cc13a24249f6c8c92816d79ec3c48d8.
Out[60]: '4d578e1acd7c670193448b84362095383cc13a24249f6c8c92816d79ec3c48d8'

But ideally it would derive a new column (ID) and add that '4d578e1acd...' value, as well. If I try to use this function on a DataFrame with > 1 row it gives me the error:

ValueError: can only convert an array of size 1 to a Python scalar

I'd like the function to be used in a lambda that can operate on a pd.DataFrame with an arbitrary number of rows, and expect output to be another pd.DataFrame with the same number of rows, but features + 1 (for the new ID column).

Is that possible? I see several similar questions, but I am not sure I can (or want to?) do this on an entire pd.Series given the function above will have some data-cleansing steps dependent on the value of partner...

pandas dataframe hash lambda python
4 Answers

You have to make a small change in order to use the function with a lambda.

function make_partnerhash(datarow, partner : str):

   h = 'a_default_value_like_Partner_has_no_hashing_strategy'

   if partner == "Partner A":
      id = datarow['ID']
      ... calculate hash etc ...
   return h

Then you can call the function from a Lambda like this:

df['HASH_COLUMN_NAME'] = df.apply(lambda x: make_parnerhash(x, 'Partner A'), axis=1)

Since you're encode() and strip() most of the columns, you can pack every field you need into a list, encode() and strip() the fields in a list comprehension, and use str.join() method to concatenate all values like this:

def make_partnerhash(row, partner: str):
    h = 'NO_HASH_DEFINED_FOR_THIS_PARTNER'
    if partner == 'Partner A':
        values_to_hash = [row['ID'],
                          pd.to_datetime(row['Date of Birth']),
                          row['Member Name'].split(",")[1]]
        
        hash_input = "".join( [ str(x).strip() for x in values_to_hash]).encode()
        h = hashlib.sha256(hash_input).hexdigest()
    
    return h
user_67603720

Consider Series.apply and avoid Series.item() to pass strings in Series into hashlib.sha256 which expects scalar values. Below translates operations to run on Series and not scalars hence the multiple calls using Series.str.

if partner == 'Partner A': 
    # CREATE PANDAS SERIES OBJECTS OR AS ADDED COLUMNS df[...] =
    health_plan_id = df['ID'].astype('str').str.strip().str.encode()
    date_of_birth = pd.to_datetime(df['Date of Birth']).astype('str').str.encode() 
    first_name = df['Member Name'].astype('str').str.split(",").str[1].str.strip().str.encode() 
    hash_input = health_plan_id + date_of_birth + first_name

    # ASSIGN NEW COLUMN OF ALL ROWS
    df['ID'] = hash_input.apply(lambda h: hashlib.sha256(string=h).hexdigest())
    return df
else:
    print("No hashing strategy defined for that partner.")
user_14224510

Adding to @Iñigo's answer, I prefer to create a callable class when a function's output depends on an external state.

class CallableHash:
    def __init__(self, partner):
        self.partner = partner
    
    def __call__(self, row):
        h = 'NO_HASH_DEFINED_FOR_THIS_PARTNER'
        if self.partner == 'Partner A':
            values_to_hash = [row['ID'],
                            pd.to_datetime(row['Date of Birth']),
                            row['Member Name'].split(",")[1]]
            
            hash_input = "".join( [ str(x).strip() for x in values_to_hash]).encode()
            h = hashlib.sha256(hash_input).hexdigest()
        # Add whatever you want to this function
        return h

You can use this in a function similar to what you've used

def make_hash(partner: str, df: pd.DataFrame) -> pd.DataFrame:
    new_df = df # Or copy.deepcopy it if you want a new df
    new_df["ID"] = new_df.apply(CallableHash(partner), axis=1)
    return new_df
user_165467060

I recommend creating a separate function for each partner
Change df to row, its work on row.
I didn't change the processing.

def make_hash_Partner_A(row) -> str:
    health_plan_id = str(row['ID']).strip().encode()
    # date_of_birth = str(dt.date(row['Date of Birth'])).encode()
    # May be an error with `dt.date`. Replaced to `pd.datetime`
    date_of_birth = str(pd.to_datetime(row['Date of Birth'])).encode() 
    first_name = str(row['Member Name']).split(",")[1].strip().encode()

    hash_input = health_plan_id + date_of_birth + first_name
    h = hashlib.sha256(string=hash_input).hexdigest()
    return h

df['hash_Partner_A'] = df.apply(make_hash_Partner_A, axis=1)
user_165383410
Related