Unique combination of two columns with mixed values

Viewed 380

I have two columns with col1 and col2

enter image description here

I'm trying to create the key column to show that a and b is the same as b and a

Problem code:

import pandas as pd
pd.DataFrame({'Col1':['a','c','b','e'],
              'Col2':['b','d','a','f']})
3 Answers

Try with frozenset

df['out'] = df.agg(frozenset,1).str.join(sep='')
Out[220]: 
0    ab
1    dc
2    ab
3    ef
dtype: object

Use numpy to sort and then join/sum the strings. This creates a unique key regardless of the order the values appear across columns.

import numpy as np

df['key'] = np.sort(df.to_numpy(), axis=1).sum(1)

#  Col1 Col2 key
#0    a    b  ab
#1    c    d  cd
#2    b    a  ab
#3    e    f  ef

In case you're interested numpy.sort is the performant way to do this. Any .agg or .apply reduce to slow loops over the rows so they will scale worse as the DataFrame becomes larger. Relative to numpy.sort the other methods are generally at least an order of magnitude slower.

import perfplot
import pandas as pd
import numpy as np
from string import ascii_lowercase


def apply_sorted(df):
    return df.apply(sorted, 1).str.join(sep='')

def numpy_sort(df):
    return pd.Series(np.sort(df.to_numpy(), axis=1).sum(1))


perfplot.show(
    setup=lambda N: pd.DataFrame({'Col1': np.random.choice([*ascii_lowercase], N), 
                                  'Col2': np.random.choice([*ascii_lowercase], N)}),
    kernels=[
        lambda df: apply_sorted(df),
        lambda df: numpy_sort(df),
    ],
    labels=['apply_sorted', 'numpy_sort'],
    n_range=[2 ** k for k in range(25)],
    equality_check=lambda x,y: x.compare(y).empty,
    relative_to=1,
    xlabel='len(df)'
)

enter image description here

I would use sorted built-in functions for this task following way

import pandas as pd
df = pd.DataFrame({'Col1':['a','c','b','e'],'Col2':['b','d','a','f']})
df['key'] = df.apply(sorted,1).str.join(sep='')
print(df)

output

  Col1 Col2 key
0    a    b  ab
1    c    d  cd
2    b    a  ab
3    e    f  ef

Explanation: it does sort every row lexicographically then join such elements. Assumption: every element of Col1 is single character and every element of Col2 is single character.

Related