Generate unique key from multiple dataframes based on name

Viewed 169

I have two data frames. As you can see, the function merges it correctly, but it is wrong. Because the carid must be unique and must not be assigned twice. How can I solve this problem? It can appear several times in a data frame, but it must remain unique over two data records. So Carid = 1 = Mercedes-benz across all data records and not Cardid = 1 = Mercedes-Benz & Citroen

import pandas as pd

d = {'Carid ': [1, 2, 3, 1], 'Carname': ['Mercedes-Benz', 'Audi', 'BMW', 'Mercedes-Benz'], 'model': ['S-Klasse AMG 63s', 'S6', 'X6 M-Power', 'Maybach']}
df = pd.DataFrame(data=d)
display(df.head())

enter image description here

d2 = {'Carid ': [4, 1, 5], 'Carname': ['VW', 'Citroen', 'Opel'], 'model': ['GTI', 'S', 'Corsa']}
df2 = pd.DataFrame(data=d2)
display(df2.head())

enter image description here

dfs = []
dfs.append(df)
dfs.append(df2)
pd.concat(dfs)

enter image description here

What I want

enter image description here

3 Answers

Method 1 Pandas Approach

First method if you don't mind changing your keys to floats is to increment using cumcount

df3 = pd.concat([df,df2])

s = df3.groupby('Carname',sort=False)['Carid'].first().to_frame()
s['Carid'] = s['Carid']  + s.groupby('Carid').cumcount() / 10

new_ids = s.to_dict(orient='dict')['Carid']

df3['Carid'] = df3['Carname'].map(new_ids)

  Carid        Carname             model
0    1.0  Mercedes-Benz  S-Klasse AMG 63s
1    2.0           Audi                S6
2    3.0            BMW        X6 M-Power
3    1.0  Mercedes-Benz           Maybach
0    4.0             VW               GTI
1    1.1        Citroen                 S
2    5.0           Opel             Corsa

Method 2 Functional Approach using Dictionaries.

Assumptions.

The logic of the function is predicated upon having a a unique carid per dataframe.

Your IDs are in a sequential order so using the max carid to generate the numbers makes the most sense. This may generate non sequential numbers if you have a list of Carids [1,2,3,200]

this would generate a new unique Carid of 201 for Citroen given that an ID of 200 already exists and is owned by a car make.

Function

import pandas as pd
import numpy as np
from collections import ChainMap


def generate_new_keys(*args,key='Carid',name='Carname'):
    """
    Takes in a number of dataframes and returns any duplicates with a new unique id. 
    groupby columns fixed to CarID and CarName.
    """
    # adds dictionaries into a single list.
    dicts_ = [arg.groupby(key)[name].first().to_dict() for arg in args]
    #merges dicts on unique key, this will exclude duplicates.
    merged_dicts = dict(ChainMap(*dicts_))
    #get the duplicate and pass the name into a list.
    delta = [v for each_dict in dicts_ for k,v in each_dict.items() if v not in merged_dicts.values()]
    # get the max sequence key
    start_key =  max(merged_dicts.keys()) + 1
    # create a new sequence
    sequence = range(start_key, start_key + len(delta) + 1)
    # return a dictionary.
    return {name : number for name,number in zip(delta,sequence)}
    

In Action

new_keys = generate_new_keys(df,df2)

print(new_keys)
{'Citroen': 6}

df3 = pd.concat([df,df2])

df3['Carid'] = np.where(df3['Carname'].isin(new_keys.keys()),
         df3['Carname'].map(new_keys), df3['Carid'])

print(df3)

   Carid        Carname             model
0    1.0  Mercedes-Benz  S-Klasse AMG 63s
1    2.0           Audi                S6
2    3.0            BMW        X6 M-Power
0    4.0             VW               GTI
1    6.0        Citroen                 S
2    5.0           Opel             Corsa

Testing additional dataframe.

new_df = pd.DataFrame({'Carid' : [1,2,3],
             'Carname' : ['Mercedes-Benz', 'Toyota','BMW'] })


new_keys = generate_new_keys(df,df2,new_df)
{'Citroen': 6, 'Toyota': 7}

df3 = pd.concat([df1,df2,new_df])

df3['Carid'] = np.where(df3['Carname'].isin(new_keys.keys()),
         df3['Carname'].map(new_keys), df3['Carid'])

print(df3)

  Carid        Carname             model
0    1.0  Mercedes-Benz  S-Klasse AMG 63s
1    2.0           Audi                S6
2    3.0            BMW        X6 M-Power
0    4.0             VW               GTI
1    6.0        Citroen                 S #< new id
2    5.0           Opel             Corsa
0    1.0  Mercedes-Benz               NaN
1    7.0         Toyota               NaN #< new id
2    3.0            BMW               NaN

One possible approach might be to do some data handling before you do the merge.

You could consider going through the smaller dataframe, and seeing what values for Carid are conflicting. Then apply new unique values for those.

I came up with this, but there's quite a lot of optimization that can be done to this:

d2 = {'Carid': [1, 2, 3, 1], 'Carname': ['Mercedes-Benz', 'Audi', 'BMW', 'Mercedes-Benz'], 'model': ['S-Klasse AMG 63s', 'S6', 'X6 M-Power', 'Maybach']}
df2 = pd.DataFrame(data=d2)

d = {'Carid': [0,1,2],'Carname': ['VW','Citroen','Opel'],'Model':['GTI','S','Corsa']}
df = pd.DataFrame(data=d)

#We loop through the bigger dataframe index values
for indx in df2.index:
    #We fetch all the ids that are present in the df
    ids = list(df2['Carid'].values)
    
    #We need this to make sure our loop doesn't break from the index size difference
    try:
        #The row we want to modify
        new_row = df.loc[indx]
        #The id it currently has
        old_id = df.Carid[indx]
        
        #Check if the id is already present 
        if old_id in ids:
            #If it is, we take the highest id out of the present ones and up it by one
            top_id = max(ids)
            new_id = top_id+1
            
            #Add it to the existing ids
            ids.append(new_id)
            
            #Set the new value
            df.Carid[indx] = new_id
        elif old_id not in ids:
            pass
        
    except KeyError as e:
        print("Index out of range")

Note that I did change the column name from Carid to just Carid

Once you've done this, there should be unqiue ids for all cars, which means you can just use concat to merge the two dataframes.

Here is my go. Note that it would be much easier if it were ok to replace Carid in both df and df2 with new i=unique ids. But staying with the question, here we go.

First we create a mapping cm between carname and carid for the first df

d = {'Carid': [1, 2, 3, 1], 'Carname': ['Mercedes-Benz', 'Audi', 'BMW', 'Mercedes-Benz'], 'model': ['S-Klasse AMG 63s', 'S6', 'X6 M-Power', 'Maybach']}
df = pd.DataFrame(data=d)
display(df.head())
cm = {name : id for name, id in zip(df['Carname'], df['Carid'])}
cm

Then we do the same for the second df

d2 = {'Carid': [4, 1, 5], 'Carname': ['VW', 'Citroen', 'Opel'], 'model': ['GTI', 'S', 'Corsa']}
df2 = pd.DataFrame(data=d2)
display(df2.head())
cm2= {name : id for name, id in zip(df2['Carname'], df2['Carid'])}
cm2

Then, the main act, combing the two mappings preserving original ids unless there is a clash in which case we assign unique ids.

unique_id = max(list(cm.values()) + list(cm2.values()))+1
for new_name in df2['Carname']:
    if new_name in cm:
        # already included
        pass
    elif cm2[new_name] not in cm.values():
        # unique carid
        cm[new_name] = cm2[new_name]
    else:
        # the new_name is not in cm but its id is

        cm[new_name] = unique_id
        unique_id += 1

print(cm)

now cm has unique id per carname, preserving the ones originally used unless they clashed:

{'Mercedes-Benz': 1, 'Audi': 2, 'BMW': 3, 'VW': 4, 'Citroen': 6, 'Opel': 5}

Now remap the ids

df['Carid'] = df['Carname'].replace(cm)
df2['Carid'] = df2['Carname'].replace(cm)

finally combine them together

dfs = []
dfs.append(df)
dfs.append(df2)
pd.concat(dfs)

the result is

|    |   Carid | Carname       | model            |
|---:|--------:|:--------------|:-----------------|
|  0 |       1 | Mercedes-Benz | S-Klasse AMG 63s |
|  1 |       2 | Audi          | S6               |
|  2 |       3 | BMW           | X6 M-Power       |
|  3 |       1 | Mercedes-Benz | Maybach          |
|  0 |       4 | VW            | GTI              |
|  1 |       6 | Citroen       | S                |
|  2 |       5 | Opel          | Corsa            |
Related